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1  Executive  Summary 

The  Air  Force  Directorate  of  Maintenance  (USAF/ILM)  and  the  Standard  Systems 
Group,  Maintenance  Systems  Division  (HQ  SSG/ILM)  sponsored  an  evaluation  of  eleven 
potential  electronic  devices  or  “E-Tools”  for  flightline  use.  The  purpose  of  this  E-Tool 
Ruggedized  Operational  Device  Evaluation  and  Observation  (RODEO)  was  to  examine 
hardware  packaging,  software  user  interface,  and  environmental  factors  associated  with  the 
usability  of  several  potential  Point  of  Maintenance  (POMx)  E-Tools  for  maintenance  data 
collection  on  the  flightline.  Air  Force  Research  Laboratory,  Logistics  Readiness  Branch 
(AFRL/HESR)  and  the  University  of  Dayton  Research  Institute  (UDRI)  were  commissioned  by 
SSG/ILM  to  independently  and  objectively  perform  the  evaluation.  The  evaluation  was 
conducted  at  the  57th  AGS,  Nellis  AFB,  Nevada,  20-22  August  2002. 

This  evaluation  employed  standard  usability  data  collection  techniques  aimed  at 
identifying  both  potential  usability  problems  and  actual  problems  with  each  of  the  devices. 
Twelve  U.S.  Air  Force  personnel  participated  in  the  study.  One  day  was  devoted  to  test 
preparation  prior  to  the  initiation  of  the  actual  test.  Testing  occurred  over  a  two-day  period,  with 
six  participants  on  the  first  day  and  six  on  the  second  day.  Participants  were  in-briefed,  given 
chemical  gear  (masks  and  gloves),  and  then  proceeded  to  the  flightline  where  they  performed  a 
series  of  actions  resembling  actual  maintenance  data  collection  activities.  Each  participant  was 
given  the  opportunity  to  use  all  eleven  devices.  Throughout  the  test,  participants  were  asked  for 
verbal  feedback.  After  using  each  device,  participants  were  asked  to  complete  a  questionnaire 
that  included  rating  scales  for  various  characteristics.  After  using  all  devices  in  a  given  category, 
participants  rank  ordered  the  devices  in  the  group  along  certain  dimensions.  Experimenters 
recorded  observations  (through  note  taking  and  video  tapes)  throughout  the  study. 

The  purpose  of  this  study  was  to  provide  a  subjective  judgment,  based  on  standard 
usability  testing  methods,  as  to  the  hardware  usability  of  various  mobile  devices  on  the  flightline. 
Based  on  analysis  of  participant  comments,  ratings  and  rankings,  the  devices  can  be  categorized 
according  to  the  following  rating  categories: 

1 .  Usable  as  designed.  Comments,  ratings  and  rankings  revealed  no  problems  or  potential 
problems. 

2.  Usable  as  currently  designed,  but  with  considerations  for  future  designs.  Comments, 
ratings  and  rankings  revealed  only  potential  problems. 

3.  Not  usable  as  currently  designed.  Comments,  ratings  and  rankings  revealed  one  or  more 
actual  problems  with  the  device  on  the  flightline. 

Using  information  collected  in  the  study,  each  device  has  been  assigned  an  appropriate 
rating  as  shown  in  the  following  matrix  (Table  I): 


Table  I.  Devices  Used  in  E-Tool  RODEO  and  Resulting  Ratings 


A.  Laptop  Devices: 

Rating 

Dolch  Computer  System 

2 

r  EDNA 

3 

ITRONIX  GoBook 

2 

Panasonic  Toughbook  28 

2 

1 


Paravant  Scorpion 

2 

B.  Handheld  Devices: 

Rating 

_ Dolphin  7400  _ 

2 

Intermec700 

2 

C.  Palm  Devices: 

Rating 

Palm  515 

2 

Svmbol  Palm 

1 

D.  Alternative  Devices: 

Rating 

LXE 

3 

Xplorer 

2 

In  summary,  9  of  the  1 1  devices  evaluated  are  suitable  for  maintenance  data  collection 
activities  on  the  flightline.  The  two  devices  not  recommended  for  use  in  their  current  state  (i.e., 
Category  3)  are  the  EDNA  and  the  LXE.  The  EDNA  had  many  complex  usability  problems 
such  as  screen  readability  and  keyboard  design.  The  LXE  could  be  made  suitable  for  flightline 
use  if  usability  issues  associated  with  the  screen  readability  can  be  resolved. 

For  those  devices  rated  in  Category  2,  several  areas  for  potential  improvement  in  future 
design  considerations  were  identified  during  testing.  Some  of  these  considerations  include. 

a.  Touch  screens  should  be  available  on  the  computing  device,  where  possible. 

b.  Multiple  means  of  entering  text  should  be  available  on  each  device  (e.g.,  hard 
keyboard,  onscreen  keyboard,  handwriting  recognition).  Where  hard  keyboards  are 
provided,  they  should  be  of  standard  QWERTY  type. 

c.  Screen  visibility  must  be  improved  for  various  lighting  conditions  on  certain 

identified  devices. 

An  informal  heat  and  sunlight  exposure  test  was  also  conducted,  see  Appendix  C. 


2 


2  Introduction 


This  document  describes  the  usability  test  conducted  by  the  Air  Force  Research 
Laboratory  and  the  University  of  Dayton  Research  Institute  to  examine  hardware  packaging, 
software  user  interface  and  environmental  factors  associated  with  the  usability  of  potential  Pom 
of  Maintenance  (POMx)  electronic  tools  (E-Tools).  The  Air  Force  Directorate  of  Mmntenance 
(USAF/ILM)  and  the  Standard  Systems  Group,  Maintenance  Systems  Division  (HQ  SSG/1L  ) 
sponsored  and  coordinated  this  usability  test  for  the  purpose  of  identifying  potential  e-tools  for 
use  at  the  point  of  maintenance.  E-tools  identified  as  usable  will  be  included  on  the  E-Tools 
Matrix-a  website  which  will  identify  e-tools  suitable  for  U.S.  Air  Force  maintenance  use. 

The  Usability  Test  evaluated  the  usability  of  1 1  potential  e-tools  for  use  in  an  Air  Force 
flightline  environment.  The  purpose  of  the  evaluation  was  to  compare  the  platformsfor 
maintenance  documentation  (e.g.,  opening  work  orders)  from  the  aircraft  location.  The  eleven 
devices  consisted  of:  1)  five  notebook  computers,  2)  two  handheld  compute^  3)  two .palm 
computers,  and  4)  two  alternative  types  (a  slate  computer  and  a  hybrid  handheld/notebook).  U.b. 
Air  Force  personnel  used  each  device  in  a  flightline-type  setting  to  simulate  opening  work 
orders.  They  were  required  to  try  each  device  with  and  without  chemical  gloves  and  masks. 

2.1  Objectives 

This  usability  test  was  designed  to  address  the  following  usability  issues: 

■  Hardware  packaging  that  deals  with  display  size. 

■  Hardware  packaging  that  deals  with  device  size.  #  . 

■  Hardware  packaging  that  deals  with  apparent  ruggedization  (perceived  ruggedization 

as  opposed  to  tested  ruggedization). 

■  Hardware  packaging  that  deals  with  keyboard  layout. 

■  Hardware  packaging  that  deals  with  pointing  devices. 

■  Software  user  interface  that  deals  with  navigation. 

■  Software  user  interface  that  deals  with  data  entry.  * 

■  Software  user  interface  that  deals  with  drop  down  menus. 

■  Environmental  factors  that  deal  with  sunlight  readability. 

■  Environmental  factors  that  deal  with  perceived  portability.  * 

■  Environmental  factors  that  deal  with  wearing  chemical  resistant  gear  (specific  issues 
addressed  for  chemical  gear  are  noted  above  with  asterisks  (*)). 


2.2  Test  Methodology 

Usability  test  methods  used  for  the  current  evaluation  are  based  on  principles  outlined  by 
Dumas  and  Redish  (1993).  In  accordance  with  usability  testing  methods  outlined  by  Dumas  & 
Redish,  design  of  the  study  included  three  steps  (see  Figure  1).  First,  the  major  usability  issues 
were  identified.  These  included  concerns  about  the  general  hardware  use  (input  devices,  size, 
and  ruggedization),  software  use  (data  entry,  and  GUI  features),  and  environmental  influences 
(sunlight,  portability,  and  use  with  chemical  gear).  From  these  general  concerns,  specific 
concerns  were  identified  (see  the  objectives  above).  Finally,  methods  for  collecting  information 
relevant  to  these  concerns  were  identified.  Methods  included  multiple  metrics  for  each  concern. 
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Figure  1.  Usability  testing  methods  (Dumas  and  Redish,  1993) 

Using  this  method,  once  the  data  is  collected,  data  analysis  leverages  the  strength  of 
triangulating  information  gathered  for  each  specific  concern.  For  example,  the  specific  concern 
dealing  with  display  size  included  use  of  three  metrics:  subject  ratings  of  the  acceptability  of  the 
screen  size,  experimenter  observations  of  the  participant  while  using  the  device,  and  specific 
participant  comments  concerning  the  size  of  the  display  (both  written  and  verbalized,  during  the 
test  and  in  post-test  discussions).  Using  triangulation,  each  metric  is  used  to  confirm  findings  o 
another  metric  so  that  if  all  three  metrics  identify  the  same  issue,  the  strength  of  usability 
concern  is  increased.  Similarly,  as  multiple  participants  identify  the  same  usability  problem,  the 
usability  problem  is  revealed.  Methods  used  to  analyze  results  are  fully  addressed  in  the  Results 
section  of  this  document. 

In  identifying  usability  concerns,  research  indicates  that  the  number  of  participants  need 
not  be  as  high  as  tests  aimed  at  determining  significance  (e.g.,  ANOVAs,  or  tests  of  correlation). 
Virzi  (1992)  identifies  that  4  to  5  subjects  identify  80%  of  the  usability  problems  with  a  system, 
and  that  additional  subjects  are  less  likely  to  identify  any  new  problems.  The  current  test 
included  12  subjects,  and  therefore  should  be  considered  to  be  of  relatively  high  strength  in 
identifying  usability  concerns  for  the  devices  tested. 
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3  Method 


3.1  Participants  and  Facility 

Twelve  U.S.  Air  Force  personnel  at  Nellis  AFB  served  as  subjects.  These  individuals 
were  assigned  to  the  57th  Aircraft  Generation  Squadron  (AGS),  the  99th  Communications 
Squadron  (CS),  the  57th  Equipment  Maintenance  Squadron  (EMS),  and  the  66th  Rescue  Squadron 
(RQS).  Their  career  backgrounds  included  both  aircraft  maintenance  and  communications 
electronics  maintenance  experience.  Testing  was  conducted  at  the  57th  AGS,  Nellis  AFB,  NV. 

3.2  Time  and  Schedule 

Pre-testing  occurred  on  the  first  day  of  testing,  August  20.  Pre-testing  consisted  of 
assuring  that  appropriate  software  was  loaded  on  all  hardware  devices.  Additionally,  scenarios 
were  all  re-evaluated  to  assure  that  the  same  types  of  manipulations  were  required  across 
devices.  Testing  was  conducted  on  August  21  &  22.  Six  subjects  participated  on  the  first  day, 
and  another  six  subjects  participated  on  the  second  day,  for  a  total  of  12  subjects.  Scheduling 
was  arranged  based  solely  on  availability,  so  as  not  to  interfere  with  subjects’  regular  work. 
Testing  on  both  days  lasted  for  most  of  the  day  (morning  and  afternoon). 

3.3  Test  Equipment  Requirements 

Hardware  used  in  the  test  included  the  items  listed  below. 

1 .  Dolch  Computer  System,  spare  battery,  charger. 

2.  EDNA  computer. 

3.  Itronix  GoBook  series,  spare  battery,  and  charger. 

4.  Panasonic  Toughbook  28  series  with  option  of  192M  RAM,  spare  battery,  and  charger. 

5.  Paravant  rugged  portable  computer,  spare  battery,  charger. 

6.  Dolphin  7400  mobile  computer,  spare  battery,  and  charger. 

7.  Intermec  710,  spare  battery,  and  charger. 

8.  Palm  515,  and  charger. 

9.  Symbol  palm,  and  charger. 

10.  LXE  computer,  spare  battery,  and  charger. 

11.  Xplorer  slate  computer,  spare  battery,  and  charger. 

3.4  Data  Collection  Equipment 

Data  collection  instruments  included  the  following  items. 

1 .  One  (1)  video  tape  recorder  with  batteries. 

2.  Blank  videocassette  tapes  (14). 

3.  Clipboards  (12). 

4.  Package  of  pens  (1). 

5.  Digital  Camera  (1). 

6.  Notebook  computer  for  subject  surveying  (1). 

7.  Digital  thermometer. 

8.  Antiseptic  wipes  to  assist  in  cleaning  chemical  gear. 

9.  Chemical  Gloves  and  Masks  (6  sets),  provided  by  Nellis  AFB.  Masks  were  cleaned  at 
the  end  of  each  day  and  glove  inserts  were  washed  overnight  for  sanitation  purposes. 
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3.5  Data  Collection  Packet 

Data  collection  packets  included  a  variety  of  forms.  Each  subject  was  given  a  complete 
packet  at  the  beginning  of  the  test  and  completed  all  forms  prior  to  the  end  of  the  test. 

1 .  In-briefing  (see  Attachment  2) 

2.  Consent  form  (see  Attachment  2) 

3.  Pre-test  questionnaire  (see  Attachment  2) 

4.  Post  condition  questionnaires  for  each  hardware  device  (see  Attachment  2) 

5.  Post-test  questionnaire  (see  Attachment  2) 

3.6  Data  Collection  Team 

The  data  collection  team  consisted  of  five  individuals:  1)  four  experimenters,  and  2)  a 
study  coordinator.  The  experimenters  provided  the  majority  of  the  interaction  with  test 
participants.  The  study  coordinator  was  responsible  for  assuring  that  scheduling  was  completed 
for  all  participants,  that  all  hardware  and  software  were  available  for  the  test,  and  that  in-briefing 
and  out-briefing  sessions  were  conducted  for  each  participant. 

3.7  Pre-Test  Requirements 

Several  activities  needed  to  be  completed  prior  to  beginning  this  test;  most  of  these 
activities  required  coordination  with  the  hosting  facility  or  with  the  vendors  supplying  the 
hardware  to  be  used  in  the  test.  Standard  Systems  Group,  Maintenance  Systems  Division  (HQ 
SSG/ILM)  assisted  with  many  of  these  pre-test  needs. 

Section  1 : 

1 .  Six  chemical  resistant  gloves  and  masks  were  required  for  the  test,  including  varying 
sizes  of  large,  medium  and  small. 

2.  Experimenters  required  at  least  a  half-day  of  viewing  the  CAMS  GUI.  This  was 
accomplished  during  the  first  day,  pre-testing. 

3.  Experimenters  required  time  to  view  the  POMx  software.  This  software  was  loaded  on 
the  handheld  devices  and  experimenters  reviewed  the  software  with  system  developers 
several  weeks  prior  to  the  test. 

4.  Experimenters  required  time  to  review  the  features  of  each  device  so  that  they  would  be 
familiar  with  each  device  prior  to  the  test.  This  was  accomplished  during  the  first  day, 
pre-testing. 

5.  A  list  of  final  hardware  to  be  tested  was  required  so  that  hardware  representatives  could 
be  present  for  the  test.  This  was  accomplished  during  the  first  day,  pre-testing. 

6.  Experimenters  created  and  finalized  scenarios  for  the  subjects  to  use  during  the  test. 
Scenarios  were  developed  several  weeks  prior  to  the  test  and  finalized  on  the  first  day, 
pre-testing. 

7.  The  devices  were  loaded  with  the  correct  software.  POMx  software  was  loaded  on  every 
handheld  prior  to  the  testing  period.  The  CAMS  GUI,  or  equivalent,  interface  was  loaded 
on  all  other  devices  during  the  first  day,  pre-testing. 

8.  Clearance  was  granted  for  experimenters  to  take  photos  at  Nellis  AFB.  Videotaping  and 
digital  photos  were  captured  during  the  test. 

9.  The  test  was  held  outside  and  tables  were  required  for  the  devices.  The  tables  served  as 
stations.  A  total  of  five  tables  were  needed.  Electrical  power  was  also  required  for 
charging  batteries  and  electrical  connection. 
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10.  Arrangements  for  an  indoor  locked  room  were  made  so  that  equipment  could  be  locked  in 
overnight. 

3.8  Test  Procedure 

Test  participants  were  provided  an  in-briefing  in  a  conference  room  setting.  The  in- 
briefing  provided  an  overview  of  the  purpose  of  the  test.  At  this  time  they  also  selected  the 
chemical  gear  they  would  be  using  for  the  test,  completed  the  consent  form,  and  completed  the 
pre-test  questionnaire.  Each  participant  was  given  a  clipboard  that  contained  all  questionnaires 
to  be  completed  in  the  test.  These  forms  were  ordered  in  the  sequence  in  which  participants  were 
to  interact  with  the  devices  (see  Table  II).  The  order  was  counterbalanced  across  participants  to 
address  any  possible  order  effects. 
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Table  II.  Subject  Matrix 
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Subject  8 
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GoBook 
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Four  tables  were  arranged  in  the  shade  next  to  the  flightline.  The  first  table  had  the  five 
notebook  computers,  the  second  table  had  the  two  handheld  computers,  the  third  table  had  the 
two  palm  computers,  and  the  fourth  table  had  the  alternative  devices  (the  slate  and  hybrid 
handheld  notebook).  One  experimenter  was  permanently  stationed  at  each  table. 

Participants  went  to  the  appropriate  table  (according  to  the  assigned  sequence).  General 
training  was  provided  so  that  basic  functions  were  learned  as  necessary  for  the  devices.  Each 
participant  started  by  taking  the  first  device  on  his  list  into  the  sunlight  and  performing  a  sample 
maintenance  open  work  order  task.  Participants  were  permitted  to  repeat  the  task  if  desired;  no 
time  or  error  data  were  collected  on  these  tasks.  Once  they  completed  the  task,  they  were  asked 
to  fill  out  a  post-condition  questionnaire  (see  Attachment  2).  At  this  point  participants  proceeded 
to  the  next  device  in  their  sequence  and  repeated  the  process.  After  having  used  all  devices  on 
the  table,  participants  were  asked  to  complete  a  post-test  questionnaire  asking  for  rank  order 
preferences  on  all  of  the  devices  at  that  table  (see  Attachment  2). 

Following  this  process,  participants  were  asked  to  don  their  chemical  gear  and  repeat 
selected  portions  of  the  process  with  the  gear  on  (e.g.,  use  each  device  in  the  sunlight).  Having 
completed  this,  participants  again  completed  a  questionnaire  on  each  device,  and  provided  rank 
orders  for  the  devices.  After  completing  the  chemical  gear  portion  of  the  test  at  each  table, 
participants  proceeded  to  the  next  table  and  began  the  process  again. 
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4  Results 


Analysis  of  the  data  collected  during  this  study  included  evaluations  of  each  device,  with 
and  without  the  use  of  chemical  gear,  and  relative  ranking  of  devices  within  a  station  along  a 
series  of  items.  These  analyses  are  detailed  in  the  following  sections.  It  is  important  to  note  that 
a  triangulation  method  was  used  in  the  analysis  of  findings  for  each  device.  Triangulation  was 
implemented  as  defined  in  the  following  paragraphs. 

Ratings  for  each  device  were  gathered  for  each  item  in  the  questionnaire  (e.g.,  for  each 
question),  and  mean  ratings  and  standard  deviations  were  then  calculated  on  each  item  for  that 
device.  Ratings  could  range  from  1  to  5  (one  was  positive  and  five  was  negative). 

A  priori,  criteria  were  set  that  1)  any  item  with  an  average  rating  of  >  3  indicated  a 
usability  problem,  and  2)  any  item  where  the  total  of  the  average  rating  plus  standard  deviation 
was  >  3  indicated  a  potential  usability  problem. 

Once  this  initial  analysis  was  complete,  user  ratings  were  plotted  on  a  cluster  graph  for 
items  that  indicated  usability  problems  or  potential  usability  problems.  User  comments  and 
observer  notes  were  then  analyzed  and  collapsed  to  assist  in  the  definition  and  clarification  of  the 
problem  or  potential  problem.  That  is,  user  comments  and  observer  notes  that  related 
specifically  to  the  item  were  added  to  the  analysis  to  assist  in  further  definition  of  the  problem  or 

potential  problem. 

For  each  usability  concern  (e.g.,  readability  in  sunlight,  use  of  pointing  device)  users 
were  asked  to  rank  order  their  preferences  across  devices.  Rankings  were  then  analyzed  to 
further  validate  the  rating  data.  If  the  device  had  usability  problems  or  potential  problems  and 
was  ranked  lower  than  the  other  devices  (less  preferred)  this  further  substantiated  that  the  device 
had  usability  issues  associated  with  it. 

4.1  Laptop  Devices 

4.1.1  Dolch  Computer  System 

The  Dolch  laptop  computer  has  a  standard  QWERTY  keyboard.  The  input  device  is  a 
keyboard  touchpad  with  two-button  control.  It  does  not  have  a  touch  screen. 

4.1. 1.1  Without  Chemical  Gear 

The  mean  and  standard  deviations  for  user  responses  to  the  acceptability  ratings  for  the 
Dolch  laptop  in  the  non-chemical  gear  condition  are  included  in  Figure  2.  No  means  or  standard 
deviations  were  in  the  unacceptable  range. 
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Laptop  Computers  -  Dolch 


Figure  2.  Means  and  Deviations  -  Dolch  Laptop  Computer 

When  users  were  not  wearing  chemical  gear,  no  problems  or  potential  problems  for  the 
Dolch  laptop  computer  were  revealed  through  user  rankings. 

Five  participants  listed  the  laptop  keyboard  as  their  favorite  feature  of  the  Dolch, 
indicating  that  the  “keyboard  keys  (were)  extremely  easy  to  use,  not  cumbersome.”  The 
apparent  ruggedization  of  the  computer  also  drew  positive  comments  from  four  participants. 

Two  subjects  also  mentioned  that  the  system  had  good  visibility  in  sunlight. 

Two  subjects  submitted  comments  indicating  a  desire  for  a  touch  screen.  Two  other 
constructive  comments  indicated  that  the  system  was  slightly  larger  than  desired  -  “not  as 
compact  as  I  would  like”  —  and  that  it  was  heavy. 

4. 1.1. 2  With  Chemical  Gear 

The  means  and  standard  deviations  for  user  responses  to  the  acceptability  ratings  for  the 
Dolch  laptop  in  the  chemical  gear  condition  are  included  in  Figure  3.  None  of  the  means  were  in 
the  unacceptable  range.  However,  the  standard  deviation  of  the  user  ratings  of  the  pointing 
device  in  the  chemical  gear  condition  indicated  a  potential  problem. 


12 


Laptop  Computers  -  Dolch  with  Chemical  Gear 
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Figure  3.  Means  and  Deviations  -  Dolch  Laptop  Computer  with  Chemical  Gear 
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Figure  4.  Individual  Ratings  for  Dolch  Laptop  Computer  with  Chemical  Gear 

Participant  ratings  of  the  Dolch  computer  indicated  a  potential  problem  using  the  pointing 
device  when  wearing  chemical  gear.  While  seven  individuals  rated  the  device  in  the  acceptable 
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range  for  this  category,  two  participants  rated  the  device  as  “Borderline,”  and  three  participants 
rated  the  access  of  the  Dolch  using  the  pointing  device  as  “Moderately  Unacceptable”  (Figure  4). 

While  one  participant  noted  that  the  “mouse  was  very  easy  to  navigate  with,  very  light,” 
five  users  noted  some  difficulty  using  the  mouse/mouse  touchpad.  The  touchpad  and  mouse 
were  listed  three  times  when  subjects  were  asked  to  classify  their  least  favorite  feature  (but 
another  subject  listed  the  touchpad  as  his  favorite  feature  on  the  device).  Other  comments 
indicated  that  the  touchpad  “almost  didn’t  work”  and  sometimes  failed  to  take  inputs,  that  the 
pointing  device  was  difficult  to  use/move  with  gloves,  and  that  a  touch  screen  was  a  necessity  for 
a  laptop  when  one  was  wearing  chemical  gear. 

Other  positive  comments  relevant  to  use  of  the  Dolch  laptop  computer  while  wearing 
chemical  gear  indicate  that  two  users  perceived  the  system  and  screen  as  “nice  and  rugged,”  and 
two  participants  liked  the  display  and  found  it  easy  to  read.  Also,  two  participants  provided 
positive  comments  about  the  laptop  keypad  keys,  as  the  keyboard  was  quite  “functional,”  and  the 
keys  were  “easy  to  get  to  with  gloves.”  Two  subjects  rated  the  keyboard  as  their  most  favorite 
feature  on  the  Dolch  laptop  computer. 

Other  constructive  comments  indicated  that,  for  two  users  in  particular,  the  screen  might 
have  faded  a  bit  in  the  sun. 

4.1.2  EDNA 

The  EDNA  laptop  computer  has  a  standard  QWERTY  keyboard.  The  input  device  is  a 
button-like  joystick  mouse.  It  does  not  have  a  touch  screen. 

The  results  for  EDNA  indicate  that  only  ten  respondents  (n=10)  were  able  to  provide 
feedback  on  this  device.  During  the  first  day  of  the  study,  connectivity  issues  on  the  device 
prevented  two  participants  from  completing  the  designed  task,  which  involved  interacting  with  a 
CAMS  data  collection  form.  As  they  did  not  complete  a  relevant  task  with  the  EDNA,  these  two 
individuals  were  unable  to  provide  feedback  on  the  device.  Subsequent  users  were  provided 
with  a  CAMS-like  interface  as  they  interacted  with  the  EDNA.  The  feedback  from  these  users  is 

reflected  below. 

4. 1.2.1  Without  Chemical  Gear 

The  means  and  standard  deviations  of  user  responses  to  the  EDNA  in  the  non-chemical 
gear  condition  are  presented  in  Figure  5.  Problems  with  computer  size,  reading  data,  and 
perceived  portability  are  highlighted  by  means  in  the  unacceptable  range.  An  additional  six 
categories  yielded  standard  deviations  in  the  unacceptable  range.  These  indicate  that  the  display 
size,  layout,  pointing  device,  entering  data,  navigating  and  drop  down  menus  on  the  EDNA  are 
potential  problems. 
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Laptop  Computers  -  EDNA 
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Figure  6.  Individual  Ratings  for  EDNA  Laptop  Computer  (continued) 

Individual  participant  responses  during  the  non-chemical  gear  condition  identified 
problems  with  the  computer  size,  reading  data,  and  perceived  portability  of  the  EDNA  laptop, 
and  revealed  potential  for  problems  with  the  display  size,  keyboard  layout,  pointing  device, 
navigating,  entering  data,  and  using  the  drop  down  menus  on  the  EDNA  laptop  (  igure  ). 

As  several  of  these  problems  and  potential  problems  may  be  related  to  one  another,  they 
will  be  addressed  in  the  most  relevant  pairings.  For  example,  participant  responses  indicated 
problems  with  computer  size  and  perceived  portability.  In  the  Computer  Size  category  onthe 
EDNA,  only  one  response  was  in  the  acceptable  range.  Three  participants  rated  this  aspect  ot  t  e 
EDNA  as  “Borderline,”  while  six  participants  rated  the  computer  size  of  the  EDNA  as 
“Moderately  Unacceptable.”  Comparatively,  three  responses  were  in  the  acceptable  range  for  ^ 
the  Perceived  Portability  category.  Four  participants  rated  this  aspect  of  EDNA  as  Borderline 
two  rated  it  “Moderately  Unacceptable”  and  one  rated  it  “Extremely  Unacceptable.  Participants 
overwhelmingly  remarked  that  the  EDNA  was  too  heavy  or  too  bulky,  with  eight  of  ten  subj  ects 
providing  negative  comments  about  the  size  and  weight  of  the  device.  It  was  not  portable,  too 
heavy  ”  “heavy  and  bulky,  won’t  fit  in  most  toolboxes,”  “way  too  bulky  in  size,”  and  simply, 
“too  big.”  Thus,  the  computer  size  -  classified  as  “too  bulky”  by  several  participants  -  would 
have  negatively  impacted  perceived  portability.  Furthermore,  two  participants  expressed  other 
portability  concerns,  pointing  out  that  the  power  cord  was  difficult  to  work  with  and  specifying 
that  they  did  not  like  “having  to  maintain  a  (power)  connection.”  The  EDNA  is  not  battery 

powered. 

While  the  size  and  weight  of  the  EDNA  negatively  impacted  participant  responses  in  die 
Computer  Size  and  Perceived  Portability  categories,  these  elements  also  may  have  impacted  the 
strong  indication  that  the  EDNA  was  perceived  as  quite  rugged.  Favorite  features  of  the  device 
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were  “the  ruggedness”  (in  two  responses),  that  it  was  “very  tough,  and  that  it  can  actually 
withstand  some  pounding!” 

Keyboard  layout,  entering  data,  navigation,  use  of  drop  down  menus,  and  ease  of  using 
the  pointing  device  were  revealed  as  potential  problems  based  upon  participant  responses.  In  the 
Keyboard  Layout  category,  six  participants  rated  the  EDNA  in  the  acceptable  range;  three 
participants  rated  it  as  “Borderline,”  and  one  participant  rated  the  device  as  Extremely 
Unacceptable  ”  Eight  participants  rated  entering  data  in  the  acceptable  range,  with  six 
individuals  rating  entering  data  on  the  EDNA  as  “Completely  Acceptable.  One  participant 
rated  entering  data  on  the  EDNA  “Moderately  Unacceptable,”  and  one  participant  rated  this 
“Extremely  Unacceptable.”  While  two  participants  provided  positive  remarks  about  the 
keyboard,  listing  the  “waterproof  keyboard”  and  “good  keyboard”  as  a  favorite  feature  of  the 
device,  three  individuals  listed  this  as  the  least  desirable  feature  of  the  EDNA,  indicating  that  the 
“keypad  was  too  tight  (you  really  have  to  press  hard).”  This  difficulty  with  the  keyboard  may 
have  been  one  facet  of  the  somewhat  negative  ratings  individuals  provided  regarding  entering 
data  on  the  EDNA. 

Seven  participants  rated  the  ease  of  using  die  pointing  device  in  the  acceptable  range, 
with  five  of  those  ratings  specifying  that  the  EDNA  was  “Completely  Acceptable  in  this 
category.  One  participant  rated  the  ease  of  using  the  pointing  device  on  the  EDNA  as 
“Borderline,”  one  participant  rated  it  as  “Moderately  Unacceptable,  and  one  participant  rated  it 
as  “Extremely  Unacceptable.”  Navigation,  an  extension  of  the  use  of  the  pointing  device,  is  a 
potential  problem,  as  well.  Eight  participants  rated  navigation  on  the  EDNA  in  the  acceptable 
range.  One  individual  provided  a  rating  of  “Borderline,”  and  one  individual  provided  a  rating  of 
“Extremely  Unacceptable”  for  the  category  of  navigation  on  the  EDNA.  For  use  of  drop  down 
menus  on  the  EDNA,  seven  participants  provided  ratings  in  the  acceptable  range.  One 
participant  provided  a  rating  of  “Borderline,”  and  one  participant  provided  a  rating  of 
“Extremely  Unacceptable.”  One  participant  did  not  provide  a  rating  for  drop  down  melius  on  the 
EDNA  in  this  condition.  Only  two  participants  provided  comments  corroborating  the  potential 
problems  specific  to  navigational  (i.e.,  pointing  device,  navigation,  and  drop  down  menus) 
issues.  One  individual  listed  the  mouse  adaptor  as  his  least  favorite  feature  of  the  device, 
another  individual  commented  that  the  device  was,  simply,  “difficult  to  maneuver. 

Two  aspects  of  the  display  on  the  EDNA  require  discussion,  as  well.  Individual 
responses  indicate  that  reading  data  on  the  EDNA  is  a  problem.  Only  two  ratings  were  in  the 
acceptable  range  in  this  category;  four  participants  provided  a  rating  of  Borderline,  one 
provided  a  rating  of  “Moderately  Unacceptable,”  and  one  provided  a  rating  of  “Extremely 
Unacceptable”  when  rating  the  ease  of  reading  data  on  the  EDNA.  In  addition,  display  size  was 
revealed  as  a  potential  problem  based  upon  individual  responses.  Five  responses  were  in  the 
acceptable  range;  however,  two  participants  rated  die  EDNA  as  Borderline,  two  participants 
rated  the  device  as  “Moderately  Unacceptable”  and  one  participant  rated  it  as  “Extremely 
Unacceptable”  in  the  Display  Size  category.  One  can  surmise  that,  in  some  cases,  these 
categories  are  related;  two  participants  indicated  they  were  unhappy  with  the  “small  screen,  and 
that  it  was  “hard  to  read,  even  worse  in  the  sun.”  Six  individuals  specified  difficulty  reading  the 
screen  in  direct  sunlight;  e.g.,  “I  needed  to  shade  the  screen  to  see  anything  ;  could  not  see 
display  in  direct  sunlight”;  “can’t  see  screen  well  with  sunglasses  on,  had  to  take  them  off  to 
complete  (the)  task.”  One  participant  simply  specified  “screen  brightness”  as  one  of  his  least 
favorite  features. 
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Other  constructive  comments  were  provided;  two  participants  indicated  that  it  is 
important  that  the  device  is  radio  frequency  connected  or  capable. 

4.1. 2.2  With  Chemical  Gear 

The  means  and  standard  deviations  for  user  responses  to  the  acceptability  ratings  for  the 
EDNA  laptop  in  the  chemical  gear  condition  are  included  in  Figure  7.  The  means  for  both 
reading  data  and  perceived  portability  are  in  the  unacceptable  range,  indicating  problems  with 
both  of  these  areas  on  the  EDNA. 


Laptop  Computers  -  EDNA  with  Chemical  Gear 


Completely  Reasonably  Borderline  Moderately  Extremely 
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Figure  7.  Means  and  Deviations  -  EDNA  Laptop  Computer  with  Chemical  Gear 
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Figure  8.  Individual  Ratings  for  EDNA  Laptop  Computer  with  Chemical  Gear 

Participant  responses  during  the  chemical  gear  condition  indicated  problems  with  reading 
data  and  with  perceived  portability  on  the  EDNA  laptop  (Figure  8).  After  using  the  device  while 
wearing  chemical  gear,  only  one  participant  rated  the  ability  to  read  data  on  EDNA  in  the  ^ 

acceptable  range.  Four  participants  rated  the  ability  to  read  data  on  the  EDNA  as  Borderline, 
two  participants  rated  the  ability  as  “Moderately  Unacceptable,”  and  three  participants  rated  the 
ability  as  “Extremely  Unacceptable.”  The  problem  with  reading  data  in  chemical  gear  is  slightly 
more  pronounced  than  it  was  in  the  regular  condition.  Six  participants  classified  the  screen  and 
its  visibility  as  a  least  favorite  feature  of  the  device,  e.g.,  “screen  unreadable,”  “hard  to  see  in 
sunlight,”  “screen  (is)  too  dark  in  sunlight”  or,  simply,  “screen.”  One  additional  participant 
specified  problems  with  this  aspect  of  the  device  with  his  more  general  comments;  “very  hard  to 
see  in  direct  sunlight  and  with  chem.  mask.” 

In  the  Perceived  Portability  category,  only  two  participants  rated  the  EDNA  in  the 
acceptable  range;  each  of  these  participants  provided  ratings  of  “Reasonably  Acceptable.  Six 
participants  rated  the  perceived  portability  of  the  device  as  “Borderline,”  one  participant  rated 
the  device  as  “Moderately  Unacceptable,”  and  one  participant  rated  the  device  as  “Extremely  ^ 
Unacceptable.”  Four  individuals  provided  comments  indicating  that  the  EDNA  was  “too  heavy 

and  “bulky”;  “it’s  like  carrying  a  sack  of  potatoes  ” 

Other  constructive  comments  related  to  the  EDNA  in  the  chemical  gear  condition 
referenced  the  keypad  of  the  device.  More  than  three  individuals  found  that  the  “buttons  worked 
well  with  gloves,”  that  the  “raised  keys  made  it  easy  to  type,”  and  that  the  “letters  are  spaced 
well,”  with  the  caveat  that  the  buttons  are  “still  too  tough  to  press.” 
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4.1.3  1TRQNIX  GoBook. 


The  Itronix  GoBook  laptop  computer  has  a  standard  QWERTY  keyboard.  The  input 
device  options  are  a  keyboard  touchpad  with  three-button  control,  or  a  touch  screen  that  can  be 
accessed  via  the  stylus.  It  weighs  approximately  seven  and  one-half  pounds. 

4. 1.3.1  Without  Chemical  Gear 

The  means  and  standard  deviations  of  user  responses  to  the  Itronix  GoBook  in  the  non¬ 
chemical  gear  condition  are  presented  in  Figure  9.  While  no  means  are  in  the  unacceptable 
range,  the  standard  deviation  in  the  Apparent  Ruggedization  category  extends  into  the 
unacceptable  range. 


Laptop  Computers  -  Itronic  GoBook 


Completely  Reasonably  Borderline  Moderately  Extremely 

Acceptable  Acceptable  Unacceptable  Unacceptable 


Figure  9.  Means  and  Deviations  -  Itronix  GoBook  Laptop  Computer 

The  compilation  of  individual  user  responses  in  the  non-chemical  gear  condition 
indicates  that  the  apparent  ruggedization  of  the  Itronix  GoBook  is  a  potential  problem  (Figure 
10). 
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Figure  10.  Individual  Ratings  for  Itronix  GoBook  Laptop  Computer 

While  six  responses  were  in  the  acceptable  range,  five  subjects  rated  the  device  as 
“Borderline”  in  regards  to  apparent  ruggedization,  while  one  participant  rated  the  device 
“Moderately  Unacceptable.”  Two  participants  remarked  that  the  Itronix  GoBook  appeared  very 
fragile  while  one  participant  commented  that  the  machine  was,  simply,  “not  rugged.”  However, 
four  participants  listed  the  size  and/or  weight  of  the  computer  as  their  favorite  feature.  It  is 
almost  everything  to  look  for:  size,  layout,  and  accessibility. 

More  generally,  two  participants  indicated  concern  regarding  the  behavior  of  the  system 
when  the  screen  was  closed  for  transportation  while  the  machine  was  still  running.  The  machine 
went  into  sleep  mode,  and  participants  were  no  longer  able  to  access  the  machine,  as  they  did  not 
possess  the  password.  Also,  two  participants  indicated  some  concern  about  the  visibility  and 
fading  of  the  screen  in  direct  sunlight.  Four  individuals  suggested  that  the  mouse  was  too 
sensitive  or  difficult  to  use  (with  two  individuals  classifying  it  as  “very  touchy”).  Specific  to  the 
keyboard,  one  participant  indicated  that  the  right  shift  button  was  too  far  over,  while  another 
participant  pointed  out  that  “keyboard  keys  tend  to  hide  behind  other  keys  when  typing. 

4. 1.3.2  With  Chemical  Gear 

The  means  and  standard  deviations  of  user  responses  to  the  Itronix  GoBook  in  the 
ohpmiral  gear  condition  are  presented  in  Figure  11.  While  none  of  the  means  are  in  the 
unacceptable  range,  the  standard  deviation  of  the  Entering  Data  category  extends  slightly  into  the 
unacceptable  range,  suggesting  the  slight  potential  for  a  problem  entering  data  on  the  Itronix 
GoBook  in  the  chemical  gear  condition. 
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Laptop  Computers  -  Itronix  GoBook  with  Chemical  Gear 
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Figure  1 1 .  Means  and  Deviations  -  Itronix  GoBook  Laptop  Computer  with  Chemical  Gear 

The  compilation  of  individual  user  responses  provides  a  very  slight  indication  that 
entering  data  on  the  Itronix  GoBook  while  wearing  chemical  gear  is  a  potential  problem.  While 
eight  participants  rated  the  device  in  the  acceptable  range  for  the  Entering  Data  category,  three 
participants  provided  a  rating  of  “Borderline,”  and  one  participant  rated  the  Itronix  GoBook 
“Moderately  Unacceptable”  (Figure  12). 
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Figure  12.  Individual  Ratings  for  Itronix  GoBook  Laptop  Computer  with  Chemical  Gear 

While  two  participants  remarked  on  the  ease  of  using  the  touch  screen  and  keyboard 
when  wearing  gloves,  and  two  other  participants  listed  the  button-style  pointing  device  as  their 
favorite  feature  on  this  device,  six  individuals  indicated  some  level  of  difficulty  with  the  mouse. 
For  one  subject,  the  “mouse  pad  (was)  a  little  awkward”;  for  another,  it  was  “hard  to  see  the 
mouse  icon  when  moving  it  around”;  another  subject  simply  stated  that  the  pointing  device  was 
unacceptable,  but  did  not  indicate  which  pointing  device  he  was  referring  to.  Furthermore,  two 
individuals  pointed  out  that  the  keyboard  keys  were  too  small  and  too  close  together  to 
effectively  use  while  wearing  chemical  gear.  It  should  be  noted  that,  as  with  other  laptop 
devices,  the  Itronix  GoBook  had  multiple  methods  for  entering  data  or  navigating  with  the 
pointing  device  (e.g.,  touch  screen,  mouse  button,  touchpad);  the  individuals  were  not  required  to 
rank  these  features  separate  from  one  another. 

More  generally,  two  participants  provided  positive  comments  indicating  that  the 
equipment  was  light  and  compact.  One  participant  listed  it  as  all-around  the  best  laptop  he 
dealt  with. 

Conversely,  two  individuals  had  trouble  with  the  screen  and  display  on  the  Itronix 
GoBook,  suggesting  that  it  may  have  been  a  bit  difficult  to  read. 

4.1.4  Panasonic  Touahbook  28 

The  Panasonic  Toughbook  28  laptop  computer  has  a  standard  QWERTY  keyboard.  The 
input  device  options  are  a  keyboard  touchpad  with  two-button  control,  or  a  touch  screen  that  can 
be  accessed  via  the  stylus.  It  weighs  approximately  nine  pounds. 
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4.1. 4.1  Without  Chemical  Gear 

The  means  and  standard  deviations  of  user  responses  to  the  Panasonic  Toughbook  in  the 
non-chemical  gear  condition  are  presented  in  Figure  13.  No  means  or  standard  deviations  are  in 
the  unacceptable  range  for  this  device  in  this  condition. 
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Figure  13.  Means  and  Deviations  -  Panasonic  Toughbook  Laptop  Computer 

When  users  were  not  wearing  chemical  gear,  no  problems  or  potential  problems  for  the 
Panasonic  Toughbook  laptop  computer  were  revealed  through  individual  user  ratings. 
Participants  provided  various  positive  comments,  including  three  participants  who  specified  th 
touch  screen  as  their  favorite  feature  on  this  device.  For  example,  after  using  the  device  in  the 
regular  condition,  one  participant  was  certain  that  “touch  screen  is  a  must  with  chem  gear 
Love  it.”  Two  participants  classified  the  size  and  weight  of  the  device  as  their  favorite  feature, 
while  another  two  participants  provided  positive  comments  about  the  rugged  design  o  e 
Panasonic  Toughbook. 

Other  comments  indicate  that  the  screen  may  have  been  difficult  to  see  in  direct  sunlight 
for  two  individuals.  Furthermore,  there  were  two  indications  of  difficulty  using  fee  mouse >  and 
touchpad.  Two  other  participants  remarked  feat,  while  fee  stylus  is  quite  useful,  it  is  also  highly 

likely  to  get  lost. 

4. 1.4.2  With  Chemical  Gear 

The  means  and  standard  deviations  of  user  responses  to  the  Panasonic  Toughbook  m  the 
chemical  gear  condition  are  presented  in  Figure  14.  While  none  of  the  means  are  in  fee 
unacceptable  range,  fee  standard  deviation  of  fee  Pointing  Device  category  extends  slightly  into 
the  unacceptable  range,  suggesting  fee  slight  potential  for  a  problem  with  fee  pointing  device  on 
fee  Panasonic  Toughbook  in  the  chemical  gear  condition. 
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Figure  14.  Means  and  Deviations  -  Panasonic  Toughbook  Laptop  Computer  with  Chemical 

Gear 
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Figure  15.  Individual  Ratings  for  Panasonic  Toughbook  Laptop  Computer  with  Chemical  Gear 

Subject  responses  during  the  chemical  gear  condition  indicated  a  potential  problem  using 
the  pointing  device  with  the  Panasonic  Toughbook  (Figure  15).  While  seven  participants  did 
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rate  the  device  in  the  acceptable  range,  four  participants  rated  the  pointing  device  as 
“Borderline,”  one  participant  rated  it  “Moderately  Unacceptable,”  and  one  participant  rated  it 
“Extremely  Unacceptable”  after  accessing  the  Panasonic  Toughbook  while  wearing  chemical 
gear.  Likewise,  the  comments  provided  regarding  the  use  of  the  pointing  device  were  both 
positive  and  negative.  Five  participants  were  very  positive  about  the  stylus,  which  they  used 
with  the  touch  screen  as  opposed  to  manipulating  the  mouse  pad  while  in  chemical  gear; 
however,  one  individual  added  the  caveat  that  the  stylus  would  quickly  get  lost.  Another 
individual  pointed  out  that  the  “pointer  was  great  because  you  don't  have  to  try  and  manipulate 
the  mouse  pad  with  bulky  gloves  on.”  Still,  six  participants  pointed  out  that  the  mouse  was 
difficult  to  use  with  chemical  gear,  and  one  participant  added  that  the  stylus  was  difficult  to  use 
as  well;  e.g.,  the  “touchpad  (was)  very  hard  to  use  with  heat  and  chem.  gear,”  the  “mouse  and 
stylus  (were)  very  poor,  difficult  to  maneuver,”  the  “mouse  pad  (was)  not  as  sensitive  to  gloves, 
or  “the  mouse  pad  was  hard  to  move  with  gloves,”  and  “touchpad  classified  as  borderline,  stylus 
as  completely  acceptable.” 

In  general,  participants  had  positive  remarks  about  the  display,  touch  screen,  and 
keyboard.  The  touch  screen,  stylus  and  large  keypad  keys  all  were  listed  twice  as  favorite 
features  on  the  device. 

Three  participants  indicated  that  they  had  difficulty  using  the  keyboard  while  wearing 
gloves,  with  one  individual  suggesting  that  a  raised  keyboard  might  be  easier  to  access  in  this 
condition. 

4.1.5  Paravant  Scorpion 

The  Paravant  Scorpion  laptop  computer  has  a  standard  QWERTY  keyboard.  The  input 
device  is  a  fingertip-joystick  with  a  two-button  control.  This  device  does  not  have  a  touch 
screen.  It  weighs  approximately  eleven  pounds. 

4.1 .5.1  Without  Chemical  Gear 

The  means  and  standard  deviations  of  user  responses  to  the  Paravant  Scorpion  in  the  non¬ 
chemical  gear  condition  are  presented  in  Figure  16.  While  none  of  the  means  are  in  the 
unacceptable  range,  the  standard  deviations  of  both  the  Pointing  Device  and  Navigation 
categories  extend  into  the  unacceptable  range,  suggesting  the  slight  potential  for  problems  with 
the  pointing  device  and  navigation  on  the  Paravant  Scorpion  in  the  non-chemical  gear  condition. 
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Laptop  Computers  -  Paravant  Scorpion 


Figure  16.  Means  and  Deviations  -  Paravant  Scorpion  Laptop  Computer 
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Figure  17.  Individual  Ratings  for  Paravant  Scorpion  Laptop  Computer 

Individual  user  responses  to  the  Paravant  Scorpion  when  not  wearing  chemical  gear 
indicated  two  potential  problem  areas:  navigation,  and  accessing  the  machine  using  the  pointing 
device  (Figure  17).  In  each  area,  this  suggestion  of  a  potential  problem  is  slight.  Ten 
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participants  rated  accessing  the  Paravant  Scorpion  with  the  pointing  device  as  in  the  acceptable 
range,  while  two  participants  rated  this  access  “Moderately  Unacceptable”  or  “Extremely 
Unacceptable.”  One  individual  specified  the  “mouse  adaptor”  as  his  least  favorite  feature  of  the 
device.  However,  few  additional  comments  were  available  to  triangulate  this  result. 

Ten  participants  also  rated  navigation  on  the  Paravant  Scorpion  in  the  acceptable  range, 
while  two  participants  rated  this  aspect  of  the  device  as  “Moderately  Unacceptable”  or 
“Extremely  Unacceptable.”  Only  one  comment  specific  to  navigation  was  recorded,  indicating 
that  the  Paravant  Scorpion  was  “easy  to  navigate.”  One  individual  who  provided  a  rating  outside 
of  the  acceptable  range  expressed  frustration  overall,  as  the  interface  in  which  he  was  working 
was  not  functioning  correctly.  This  software  interface  malfunction  —  not  specific  to  the  device 
itself-  made  it  somewhat  difficult  for  this  user  to  successfully  navigate  through  the  program. 

Two  participants  commented  positively  on  the  fact  that  the  keyboard  appeared  water 
resistant.  Two  additional  participants  provided  positive  comments  about  the  mouse  and  easy 
navigation  capability.  Other  favorable  comments  were  recorded  regarding  good  screen  visibility 
in  direct  sunlight. 

Participants  overwhelmingly  indicated  that  the  device  was  too  heavy  and/or  bulky,  with 
seven  of  twelve  participants  specifying  this  complaint  in  their  responses. 

4. 1.5.2  With  Chemical  Gear 

The  means  and  standard  deviations  of  user  responses  to  the  Paravant  Scorpion  in  the 
chemical  gear  condition  are  presented  in  Figure  18.  No  means  or  standard  deviations  are  in  the 
unacceptable  range  for  this  device  in  this  condition. 


Laptop  Computers  -  Paravant  Scorpion  with  Chemical  Gear 
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Figure  18.  Means  and  Deviations  -  Paravant  Scorpion  Laptop  Computer  with  Chemical  Gear 
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In  the  chemical  gear  condition,  the  Paravant  Scorpion  did  not  show  any  problems  or 
potential  problems.  Participants  responded  particularly  favorably  to  the  keyboard  layout  and  its 
ease  of  use;  five  participants  classified  the  keyboard  and  keys  as  well  spaced,  of  good  size  and 
height,  and  easy  to  maneuver  when  wearing  chemical  gear.  It  should  be  noted  that  one 
participant  did  remark  that  he  had  difficulty  with  the  keyboard,  stating  that  he  had  to  hold  (his) 
hands  away  so  (he)  wouldn’t  bump  other  keys.” 

4.1.6  Rankings  for  Laptop  Devices 

Statistical  analysis  of  the  rankings  may  show  if  one  laptop  device  is  preferred  over  the 
others.  Using  the  x  2  analysis,  the  analysis  is  done  on  the  “Most  Preferred”  rankings  provided  by 
participants.  If  the  distribution  is  equal,  the  expected  value  for  each  configuration  is  2.4  (n  —  12). 
With  four  (4)  degrees  of  freedom  (d /  =  k-1)  and  a  =  .05,  the  critical  value  is  9.488. 

In  the  non-chemical  gear  condition,  the  calculation  yields  x  2  =  5.501 .  Since  x  2  <  9.488, 
the  null  hypothesis  cannot  be  rejected  and  therefore  the  data  did  not  provide  significant  evidence 
that  the  subjects  had  a  statistical  preference  for  a  particular  laptop  device  in  the  non-chemical 
gear  condition.  In  the  chemical  gear  condition,  the  calculation  yields  x  =  3.833.  Since,  again, 

X  2  <  9.488,  the  null  hypothesis  cannot  be  rejected  and  therefore  the  data  did  not  provide 
significant  evidence  that  the  subjects  had  a  statistical  preference  for  a  particular  laptop  device  in 
the  chemical  gear  condition.  Even  though  this  factor  is  not  statistically  significant  in  either 
condition,  the  distribution  is  not  as  strong  due  to  the  study’s  small  sample  size;  thus,  other 
analysis  methods  were  used  to  evaluate  device  preferences. 

Overall,  visual  inspection  of  ranking  data  for  the  laptop  devices  substantiates  findings 
from  the  rating  data;  while  no  device  was  clearly  defined  as  the  most  preferred,  users  least 
preferred  the  EDNA  device  when  not  wearing  chemical  gear  (see  Figure  19)  and  while  wearing 
chemical  gear  (see  Figure  20). 
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Figure  19.  Rankings  for  Laptop  Devices 
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Figure  20.  Rankings  for  Laptop  Devices  With  Chemical  Gear 
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4.2  Handheld  computers 

4.2.1  Dolphin  7400 

The  Dolphin  7400  handheld  computer  has  an  onscreen  standard  QWERTY  keyboard,  and 
an  alphanumeric  hard  keypad.  The  input  device  options  are  the  keypad,  or  a  touch  screen  that 
can  be  accessed  via  the  stylus.  The  device  weighs  approximately  one  and  one-half  pounds. 

4.2.1. 1  Without  Chemical  Gear 

The  means  and  standard  deviations  of  user  responses  to  the  Dolphin  7400  in  the  non¬ 
chemical  gear  condition  are  presented  in  Figure  21 ,  While  none  of  the  means  are  in  the 
unacceptable  range,  note  that  the  standard  deviations  of  the  Layout  and  Entering  Data  categories 
extend  into  the  unacceptable  range,  suggesting  a  varying  potential  for  problems  with  these 
features  on  the  Dolphin  7400  in  the  non-chemical  gear  condition. 
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Figure  21.  Means  and  Deviations  -  Dolphin  7400  Handheld  Computer 
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Figure  22.  Individual  Ratings  for  Dolphin  7400  Handheld  Computer 

Participant’s  scaled  responses  in  the  non-chemical  gear  condition  indicate  potential 
problems  with  both  the  Keyboard  Layout  and  Entering  Data  categories  on  the  Dolphin  7400. 

The  trend  of  user  responses  supports  that  the  keyboard  layout  has  the  stronger  potential  for 
problems.  After  accessing  the  device,  only  five  total  participants  rated  the  keyboard  layout  on 
the  Dolphin  7400  in  the  acceptable  range.  Five  participants  rated  the  keyboard  layout  of  the 
device  “Borderline,”  while  one  participant  rated  it  “Moderately  Unacceptable,”  and  one 
participant  rated  it  “Extremely  Unacceptable”  (Figure  22). 

The  strong  indication  of  the  scaled  responses  is  further  validated  by  the  comments 
provided  by  the  participants.  Eight  participants  provided  negative  comments  about  the  hard 
keyboard  on  the  Dolphin  7400.  “It’s  too  hard  to  type  with  the  keyboard. .  .(for  example,  for)  the 
backspace  function  you  shouldn't  have  to  repress  the  red  button  and  backspace  every  time.” 
“Selection  on  the  manual  keyboard  was  difficult.”  “Don't  like  the  function  keys  for  the  manual 
keypad.” 

The  indication  of  difficulty  entering  data  on  the  Dolphin  7400  is  less  pronounced.  Nine 
participants  rated  entering  data  on  this  device  in  the  acceptable  range.  Only  three  ratings  were  in 
the  unacceptable  range;  one  participant  rated  the  device  as  “Borderline”  when  entering  data,  and 
two  participants  rated  it  “Moderately  Unacceptable.”  It  should  be  noted  that  the  strong 
indication  that  keyboard  layout  on  the  Dolphin  7400  posed  a  potential  problem  could  have  some 
impact  on  the  perceived  ease  of  entering  data  on  the  device. 

Four  participants  also  provided  some  negative  response  about  the  Dolphin  7400  on¬ 
screen  keyboard,  as  well.  While  this  keyboard  option  appeared  preferable  to  the  hard  keyboard, 
it  was  “a  bit  awkward  to  get  to.”  Other  feedback  indicated  that  using  the  on-screen  keyboard 
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“was  time  consuming”  and  that  there  was  concern  that  the  “onscreen  keyboard  would  block  my 
view  of  the  entry  fields.” 

In  general,  positive  comments  from  some  participants  indicate  that  the  size  of  the  device 
and  the  holding  strap  were  popular  features  of  the  device.  Two  participants  specified  that  they 
liked  the  size  of  the  device.  Also,  two  participants,  when  specifying  their  favorite  feature  of  the 
device,  listed  the  handheld  strap  of  the  Dolphin  7400. 

The  stylus  was  also  listed  as  a  favorable  feature  on  the  device,  as  three  participants 
indicated  a  preference  for  die  stylus  and  touch  screen.  However,  three  participants  pointed  out 
that  the  stylus  “does  not  have  a  holding  device,”  and  “. .  .should  fit  into  unit  so  it  is  not  easily 
lost.” 

4.2.1. 2  With  Chemical  Gear 

The  means  and  standard  deviations  of  user  responses  to  the  Dolphin  7400  in  the  chemical 
gear  condition  are  presented  in  Figure  23.  No  means  or  standard  deviations  are  in  the 
unacceptable  range  for  this  device  in  this  condition. 
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Figure  23.  Means  and  Deviations  -  Dolphin  7400  Handheld  Computer  with  Chemical  Gear 

User  scaled  responses  indicate  no  problems  or  potential  problems  for  the  Dolphin  7400  in 
the  chemical  gear  condition.  However,  open-ended  responses  suggest  individual  problems  with 
the  hard  keypad  on  the  Dolphin  7400  in  the  chemical  gear  condition  (this  is  validated  by  the 
scaled  responses  to  the  Dolphin  in  the  non-chemical  gear  condition).  Six  participants 
specifically  commented  that  the  hard  keypad  was  difficult  to  use  when  wearing  gloves.,  For 
example,  “you  wouldn’t  be  able  to  type  any  info  in  on  hard  keypad  while  using  gloves.”  Three 
of  these  participants  felt  that  use  of  chemical  gear  made  the  hard  keypad  impossible  to  use; 
however,  two  others  indicated  that  the  hard  keypad  was  still  acceptable  or  decent,  even 
while  the  “manual  keyboard  could  be  slightly  more  user  friendly.  Two  participants  indicated 
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that,  when  wearing  chemical  gloves,  the  stylus  was  “awkward*  or  hard  to  handle.  In  addition, 
two  participants  constructively  pointed  out  that  the  device  did  not  appear  to  have  a  stylus  storage 
location.  Given  these  comments,  it  is  notable  that  the  numeric  rating  responses  did  not  indicate 
problems  with  the  Dolphin  7400  in  the  chemical  gear  condition. 

4.2.2  Intermec  700 

The  Intermec  700  handheld  computer  has  an  onscreen  standard  QWERTY  keyboard,  and 
19-button  hard  keypad.  The  input  device  options  are  the  keypad,  or  a  touch  screen  that  can  be 
accessed  via  the  stylus.  The  device  weighs  approximately  one  pound. 

4.2.2. 1  Without  Chemical  Gear 

The  means  and  standard  deviations  of  user  responses  to  the  Intermec  700  in  the  non¬ 
chemical  gear  condition  are  presented  in  Figure  24.  While  none  of  the  means  are  in  the 
unacceptable  range,  note  that  the  standard  deviations  for  the  Entering  Data  and  Layout  categories 
extend  into  the  unacceptable  range,  suggesting  the  potential  for  problems  entering  data  and  with 
the  layout  of  the  Intermec  700  in  the  non-chemical  gear  condition. 


Handheld  Computers  -  Intermec  700 


Figure  24.  Means  and  Deviations  —  Intermec  700  Handheld  Computer 
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Figure  25.  Individual  Ratings  for  Intermec  700  Handheld  Computer 

Participant’s  scaled  responses  in  the  non-chemical  gear  condition  indicate  potential 
problems  both  with  the  keyboard  layout  and  with  entering  data  on  die  Intermec  (Figure  25). 

While  eight  participants  rated  the  keyboard  layout  on  the  Intermec  in  the  acceptable  range,  three 
participants  rated  it  as  “Moderately  Unacceptable,”  and  one  participant  indicated  thatitwas 
“Extremely  Unacceptable.”  Nine  participants  specifically  commented  that  they  had  difficulty 
with  or  simply  did  not  like  the  hard  keyboard  on  the  Intermec,  indicating  that  the  hard  keyboard 
was  “confusing  and  a  pain,”  “usable  but  confusing,”  or  simply  “not  user  friendly.”  Participants 
indicated  that  the  “phone-style  keyboard”  was  not  an  efficient  way  of  entering  alphabetic 
characters;  “die  manual  keypad  was  very  inadequate.”  One  participant  specified,  I  didn  t  like 

that  I  had  to  switch  CAPS  lock  for  features  other  than  capital  letters.” 

The  indication  of  potential  problems  with  entering  data  on  the  Intermec  is  less  extreme, 
but  still  present.  Nine  ratings  of  “Completely  Acceptable”  or  “Reasonably  Acceptable”  were 
assigned  for  this  aspect  of  the  Intermec.  However,  four  participants  rated  data  entry  on  this 
device  in  the  unacceptable  range.  There  were  two  “Borderline”  responses,  one  “Moderately 
Unacceptable”  and  one  “Extremely  Unacceptable”  response.  One  individual  supplied  two 
ratings  for  the  Entering  Data  category  for  the  Intermec;  he  indicated  that  entering  data  using  the 
stylus  was  “Reasonably  Acceptable,”  while  entering  data  using  the  hard  keyboard  was 
“Extremely  Unacceptable.”  It  appears  the  other  responses  may  also  be  a  reflection  of  the 
difficulty  individuals  experienced  and  expressed  with  the  hard  keyboard.  Only  one  participant 
indicated  that  the  “screen,  numeric  keypad  is  time  consuming”  -  most  likely  suggesting  that  both 
the  numeric  and  on-screen  keyboard  were  time  consuming.  Four  participants  specified  via 
comments  that  the  onscreen  keyboard  was  more  acceptable,  with  three  of  these  individuals 
listing  the  onscreen  keyboard  -  and  the  ability  to  access  it  via  the  stylus  -  as  their  favorite 

feature. 
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Other  positive  comments  indicated  the  participants  were  pleased  with  the  size  of  the 
device.  More  than  two  individuals  remarked  favorably  about  the  overall  size  of  the  Intermec, 
including  that  the  device  had  a  “nice  shape  and  size.” 

4.2.2.2  With  Chemical  Gear 

The  means  and  standard  deviations  of  user  responses  to  the  Intermec  700  in  the  chemical 
gear  condition  are  presented  in  Figure  26.  While  none  of  the  means  are  in  the  unacceptable 
range,  note  that  the  standard  deviation  of  the  Entering  Data  category  extends  slightly  into  the 
unacceptable  range,  suggesting  the  slight  potential  for  a  problem  entering  data  on  the  Intermec 
700  when  wearing  chemical  gear. 
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Figure  26.  Means  and  Deviations  -  Intermec  700  Handheld  Computer  with  Chemical  Gear 
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Figure  27.  Individual  Ratings  for  Intermec  700  Handheld  Computer  with  Chemical  Gear 

Participant’s  scaled  responses  further  indicate  a  potential  problem  entering  data  with  the 
Intermec  while  wearing  chemical  gear.  While  six  participants  rated  entering  data  on  the 
Intermec  in  the  acceptable  range,  six  participants  rated  this  aspect  of  the  device  in  the 
unacceptable  range,  including  two  ratings  of  “Moderately  Unacceptable”  and  one  of  “Extremely 
Unacceptable.”  Four  participants  rated  entering  data  on  the  Intermec  700  as  “Borderline” 

(Figure  27). 

From  the  participant  comments,  it  is  probable  that  the  difficulty  entering  data  was 
primarily  due  to  issues  with  the  hard  keyboard.  Ten  participants  commented  that  the  hard 
keyboard  was  difficult  to  use,  e.g.,  “keypad  was  extremely  unacceptable  in  entering  data”; 

“forget  using  the  hard  keyboard  (for)  entering  data”;  “did  not  like  the  function/letter  keys. .  .(for) 
letter  B,  push  twice”;  “the  keyboard  (manual)  is  not  user  friendly.  I  don’t  like  it!” 

Where  there  are  positive  comments  about  entering  data,  either  use  of  the  stylus  and/or  use 
of  the  on-screen  keyboard  is  implied.  One  participant  remarked  that  “entering  data  (is) 
reasonably  acceptable  as  long  as  you  use  pointer.”  Another  participant  pointed  out  that  the 
“Screen  keyboard  (is)  fast  and  easy  to  use.” 

Six  participants  provided  positive  remarks  specific  to  the  stylus.  “The  stylus  is  great, 
navigation  is  very  easy.  Good  job!”  “When  in  chemical  gear  you  can  use  the  pen  to  do 
everything.”  It  should  be  noted  that  two  participants  indicated  difficulty  with  the  size  of  the 
stylus,  as  it  was  “hard  to  grip”  and  difficult  to  remove  from  the  pen  storage. 

Other  collaborated  comments  show  that  two  users  specified  that  they  were  quite  pleased 
with  the  size  of  the  Intermec,  listing  the  size  of  the  device  as  their  favorite  feature. 
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4.2.3  Rankings  for  Handheld  Devices 

Statistical  analysis  of  the  rankings  may  show  if  one  handheld  device  is  preferred  over  the 
others.  Ranking  data  for  die  handheld  computers  was  analyzed  within  the  context  of  the 
usability  issues  most  noted  for  these  devices:  keyboard  layout,  and  entering  data.  Of  particular 
interest  in  regards  to  these  issues  are  the  variances  in  preferences  for  the  handheld  devices  when 
rating  them  for  use  entering  alphabetic  data,  and  when  rating  the  devices  for  use  entering 
numeric  data.  Using  the  x 2  analysis,  the  analysis  is  done  on  the  “Most  Preferred  rankings 
provided  by  participants  across  these  categories. 

In  the  chemical  gear  condition,  the  following  analysis  was  done  to  determine  statistical 
significance  of  preference  when  entering  alphabetic  data  on  the  handheld  devices.  If  the 
distribution  is  equal,  the  expected  value  for  each  configuration  is  5.5  (n  =  1 1).  Note  that  only 
eleven  participants  provided  accurate  rankings  for  both  devices  regarding  entering  alphabetic 
data  in  the  chemical  gear  condition.  With  one  (1)  degree  of  freedom  (d /  =  k-1)  and  a  =  .05,  the 
critical  value  is  3.8414. 

In  the  chemical  gear  condition,  the  calculation  yields  x  2=  .8181.  Since  x  <  3.8414,  the 
null  hypothesis  cannot  be  rejected  and  therefore  the  data  did  not  provide  significant  evidence  that 
the  subjects  had  a  statistical  preference  for  a  particular  handheld  device  when  entering  alphabetic 
data  in  the  chemical  gear  condition.  In  the  non-chemical  gear  condition,  the  effect  reflects  these 
results  found  in  the  chemical  gear  condition.  If  the  distribution  is  similar  across  responses  to  the 
device,  the  expected  value  for  each  configuration  is  6  (n  =  12).  In  the  non-chemical  gear 
condition,  the  calculation  yields  %2=  .3333.  Since,  again,  x 2<  3.8414,  the  null  hypothesis 
cannot  be  rejected  and  therefore  the  data  did  not  provide  significant  evidence  that  the  subjects 
had  a  statistical  preference  for  a  particular  handheld  device  when  entering  alphabetic  data  in  the 
non-chemical  gear  condition. 

When  entering  numeric  data  on  the  handheld  device  in  the  chemical  gear  condition,  the 
expected  value  for  each  configuration  is  6  (n  —  12).  The  calculation  yields  x  Since  x  < 
3.8414,  the  null  hypothesis  cannot  be  rejected  and  therefore  the  data  did  not  provide  significant 
evidence  that  the  subjects  had  a  statistical  preference  for  a  particular  handheld  device  when 
entering  numeric  data  in  the  chemical  gear  condition.  Likewise,  when  entering  numeric  data  on 
the  handheld  device  in  the  non-chemical  gear  condition,  the  expected  value  for  each 
configuration  is  6.5  (n  =  13).  The  calculation  yields  x  2  =  -692.  Since,  again,  % 2  <  3.8414  the 
null  hypothesis  cannot  be  rejected  and  therefore  the  data  did  not  provide  significant  evidence  that 
the  subjects  had  a  statistical  preference  for  a  particular  handheld  device  when  entering  numeric 
data  in  the  non-chemical  gear  condition. 

Even  though  this  factor  is  not  statistically  significant  in  any  of  the  above  conditions,  it 
should  be  noted  that  the  distribution  is  not  as  strong  due  to  the  study’s  small  sample  size;  thus, 
other  analysis  methods  were  used  in  evaluating  device  preferences. 

To  further  evaluate  any  possible  device  preferences,  ranking  data  for  the  handheld 
computers  was  visually  inspected  within  the  context  of  the  usability  issues  most  noted  for  these 
devices:  keyboard  layout  and  entering  data.  Both  the  Dolphin  and  Intermec  devices  indicated 
potential  usability  problems  in  these  areas.  Close  inspection  of  the  ranking  data  indicates  that 
users  slightly  preferred  the  Dolphin  for  alphabetical  entry  of  data  (Figure  28),  while  they  slightly 
preferred  the  Intermec  for  numeric  entry  (Figure  29).  While  statistical  differences  were  not 
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found,  it  is  interesting  to  note  that  the  Intermec  used  a  telephone  keypad  arrangement  (numerical 
with  alpha  characters  available  only  with  multiple  key  presses)  and  the  Dolphin  had  an 
alphabetically  arranged  keyboard.  These  differences  may  account  for  the  change  in  preference 
rankings.  Note  that  no  real  preferences  were  found. 


Rankings  for  Alphabetic  Data  Entry  with  Handheld  Devices 
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Figure  29.  Rankings  for  Numeric  Data  Entry  with  Handheld  Devices 

4.3  Palm  devices 

4.3.1  Palm  515 

The  Palm  515  palm  computer  has  an  onscreen  standard  QWERTY  keyboard,  and 
handwriting  recognition.  The  input  device  option  is  a  touch  screen  that  can  be  accessed  via  the 
stylus.  The  device  weighs  approximately  five  ounces. 

4.3.1. 1  Without  Chemical  Gear 

The  means  and  standard  deviations  of  user  responses  to  the  Palm  515  in  the  non-chemical 
gear  condition  are  presented  in  Figure  30.  None  of  the  means  are  in  the  unacceptable  range. 
However,  the  standard  deviations  in  the  Apparent  Ruggedization,  Pointing  Device,  Entering 
Data,  and  Drop  Down  Menu  categories  extend  (to  varying  degrees)  into  the  unacceptable  range, 
suggesting  potential  for  problems  with  these  categories  on  the  Palm  515  in  the  non-chemical  gear 
condition. 
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navigation,  ease  of  entering  data,  and  use  of  drop  down  menus  (Figure  31).  The  responses  to 
each  of  these  indicate  varying  degrees  to  which  they  may  be  problems. 

As  only  five  participants  rated  the  apparent  ruggedization  of  the  Palm  5 1 5  in  the 
acceptable  range,  the  response  does  indicate  reason  for  concern.  Four  individuals  rated  the 
apparent  ruggedization  of  the  device  as  “Borderline,”  and  three  rated  it  Moderately 
Unacceptable.”  When  asked  about  the  apparent  ruggedization  of  the  device,  four  participants 
remarked  that  the  Palm  515  was  too  small  or  fragile.  It  “does  not  feel  like  it  could  take  much 
flightline  abuse.”  “It  isn't  tough  enough  for  crew  chiefs.”  The  “screen  will  need  a  scratch 
protector.” 

It  should  be  noted  that,  overall,  the  size  of  the  device  still  received  the  most  positive 
response;  five  participants  listed  size  as  their  favorite  feature  of  the  Palm  515.  It  could  be 
transported  aircraft  to  aircraft  without  being  cumbersome,”  and  had  excellent  extra  features  (e.g., 
date  book,  notepad).  While  one  participant  was  impressed  that  the  “device  (was)  very  small  and 
easy  to  transport,”  another  subject  did  indicate  that  the  small  size  could  increase  the  likelihood 
that  the  device  may  be  lost  or  stolen. 

The  trend  of  the  user  responses  regarding  the  pointing  device  on  the  Palm  515  clearly 
indicates  a  potential  problem  in  this  area.  While  six  participants  rated  the  device  in  the 
acceptable  range,  two  participants  rated  the  Palm  515  pointing  device  as  “Borderline,”  three 
rated  it  “Moderately  Unacceptable,”  and  one  rated  it  “Extremely  Unacceptable.”  The  individual 
comments  support  the  provided  ratings.  Six  participants  reported  some  difficulty  with  the  small 
pointing  device.  While  it  was  unclear  whether  the  difficulty  was  due  to  the  stylus  or  to  the  user 
interface  responsiveness,  “certain  options  were  very  difficult  to  enter  data  into.  Thus,  various 
users  classified  this  as  difficulty  with  the  pointing  device,  while  others  classified  this  as  a  data 
entry  issue.  Other  user  comments  in  this  area  include: 

■  Small  pointing  device,  some  entries  didn't  want  to  work. 

■  Sometimes  system  did  not  like  to  respond  to  pen/stylus. 

■  Had  a  hard  time  using  pointing  device. 

■  Screen  did  not  respond  well,  had  to  search  for  points  to  hit. 

■  Some  things  were  small  and  hard  to  read  —  pointer  was  difficult  to  get  to  register. 

The  user  responses  also  show  a  slight  indication  that  navigation  on  the  Palm  515  may  be 
a  potential  problem.  Nine  participants  did  rate  the  device  in  the  acceptable  range  in  this  area, 
however,  the  remaining  three  respondents  indicated  that  the  device  was  Borderline, 

“Moderately  Unacceptable,”  and  “Extremely  Unacceptable  (one  response  in  each  category). 
Very  few  comments  directly  triangulate  this  result.  One  user  commented  that,  while  the  on¬ 
screen  keyboard  was  not  a  problem,  he  did  not  appreciate  the  navigation  feature  that  required 
switching  to  different  screens  to  enter  numeric  or  alphabetic  text.  However,  as  navigation  is  a 
reflection  of  use  of  the  pointing  device,  entering  data,  and  access  to  drop  down  menus,  the  fact 
that  potential  problems  are  indicated  in  each  of  these  areas  strengthens  the  Navigation  category 
rating  response  results. 

Related  to  navigation,  the  remaining  categories  wherein  potential  problems  are  indicated 
support  the  above  concept.  Seven  participants  indicated  that  entering  data  on  the  Palm  515  was  ^ 
in  the  acceptable  range;  however,  four  individuals  rated  this  aspect  of  the  device  as  “Borderline,” 
and  one  participant  rated  entering  data  on  the  Palm  5 1 5  as  “Extremely  Unacceptable. 
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Likewise,  eight  participants  rated  the  drop  down  menus  on  the  Palm  5 1 5  in  the  acceptable 
range,  but  one  participant  rated  them  as  “Borderline,”  and  one  participant  rated  the  drop  down 
menus  on  the  device  as  “Extremely  Unacceptable.”  It  should  be  noted  that  two  individuals  did 
not  provide  ratings  for  the  drop  down  menus  on  the  Palm  515  in  this  condition;  but  both 
commented  that  they  had  difficulty  with  the  drop  down  menus  —  e.g.,  “hard  time  with  getting 
drop  downs  to  work,”  and  “some  drop  down  options  did  not  work.”  Thus,  while  some  users 
appreciated  the  availability  of  drop  down  options  and  the  ease  with  which  text  was  subsequently 
replaced,  there  was  some  difficulty  with  the  drop  down  options  on  this  device.  However,  what 
subjects  had  a  particularly  difficult  time  with  was  selecting  and  changing  the  characters  in  the 
time  blocks  of  the  application.  “The  drop  down  menu  is  hard  to  access.  Could  not  put  in  times 
with  stylus.”  Also,  one  user  indicated  a  “difficulty  in  putting  (the)  date.  Date  block  needs  to  be 
in  bold.  Drop  down  option  is  great,  don't  have  to  type  it  in.”  This  action  required  the  selection 
of  the  text  to  be  changed;  arrows  to  adjust  various  numbers  could  only  be  used  once  the  text  was 
properly  selected.  Three  of  the  four  comments  that  indicate  difficulty  with  drop  down  menus 
specify  these  date-time  blocks.  Very  little  difficulty  was  observed  with  pure  drop  down 
functionality  on  the  Palm  515  handheld. 

4.3.1. 2  With  Chemical  Gear 

The  means  and  standard  deviations  of  user  responses  to  the  Palm  515  in  the  chemical 
gear  condition  are  presented  in  Figure  32.  None  of  the  means  are  in  the  unacceptable  range. 
However,  note  that  the  standard  deviation  of  the  Pointing  Device  category  extends  slightly  into 
the  unacceptable  range,  suggesting  the  slight  potential  for  a  problem  with  the  pointing  device  on 
the  Palm  5 1 5  in  the  chemical  gear  condition. 


Palm  Computers  -  Palm  515  with  Chemical  Gear 


Completely  Reasonably  Borderline  Moderately  Extremely 

Acceptable  Acceptable  Unacceptable  Unacceptable 


Figure  32.  Means  and  Deviations  of  Palm  515  Palm  Computer  with  Chemical  Gear 
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Figure  33.  Individual  Ratings  for  Palm  515  Palm  Computer  with  Chemical  Gear 

User  responses  to  the  Palm  5 1 5  in  the  chemical  gear  condition  indicate  that  using  the 
pointing  device  while  wearing  chemical  gear  posed  a  potential  problem.  While  eight  individuals 
rated  the  device  in  the  acceptable  range,  three  participants  rated  the  device  Borderline  and  one 
participant  rated  the  device  “Moderately  Unacceptable”  when  considering  access  to  the  Palm  515 
using  the  pointing  device  (Figure  33).  Six  participants  listed  the  stylus  as  their  least  favorite 
feature  of  the  Palm  515  in  the  chemical  gear  condition.  Overall,  eight  participants  commented  in 
some  negative  way  on  the  use  of  the  stylus  in  the  chemical  gear  condition;  the  stylus  was 
awkward  to  retrieve  and  difficult  to  hold  when  wearing  chemical  gloves,  and,  at  times,  it  was 
somewhat  inefficient  as  a  data  entry  tool  (particularly  in  the  sensitive  date  and  time  blocks).  One 
user  indicated  some  concern  that  the  device  itself  was  too  small  to  work  with  while  wearing 
chemical  gear. 

4.3.2  Symbol  Palm 

The  Symbol  palm  computer  has  an  onscreen  standard  QWERTY  keyboard,  and 
handwriting  recognition.  The  input  device  option  is  a  touch  screen  that  can  be  accessed  via  the 
stylus.  The  device  weighs  between  10  and  13  ounces,  depending  on  configuration. 

4.3.2. 1  Without  Chemical  Gear 

The  means  and  standard  deviations  of  user  responses  to  the  Symbol  palm  computer  in  the 
non-chemical  gear  condition  are  presented  in  Figure  34.  Note  that  all  of  the  means  and  standard 
deviations  are  in  the  acceptable  range. 
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Palm  Computers  -  Symbol 
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Figure  34.  Means  and  Deviations  -  Symbol  Palm  Computer 
When  users  were  not  wearing  chemical  gear,  no  problems  or  potential  problems  for  the 

Symbol  Palm  device  were  revealed  through  user  rankings. 

Multiple  positive  comments  indicate  that  the  device  was  quite  easy  to  use;  one  participant 
remarked  that  the  Symbol  Palm  is  a  “very  good  device,  very  user  friendly.”  Three  participants 
simply  listed  the  overall  ease  of  use  as  their  favorite  feature  of  the  device,  while  one  simply 
pointed  out,  “I  found  nothing  that  I  dislike.”  Two  participants  remarked  positively  on  the 
portability  of  the  device.  Users  expressed  that  they  envisioned  using  this  device  primarily  for 
tech  data  or  checklists,  as  opposed  to  using  it  for  opening  and  closing  CAMS  records. 

Constructive  comments  regarding  the  Symbol  Palm  indicate  that  two  participants  may 
have  had  minor  difficulty  selecting  items  -  one  participant  remarked  that,  when  selecting  one 
item,  the  item  adjacent  to  it  occasionally  was  selected  instead,  and  another  remarked  that  (as  on) 
“the  Palm  515,  the  letter  and  number  keypad  icons  should  be  bigger”  -  and  were  somewhat 
uncertain  about  toggling  the  onscreen  keypad  to  access  both  letters  and  numbers.  This  is  true  for 

both  palm  devices. 

4.3.2.2  With  Chemical  Gear 

The  means  and  standard  deviations  of  user  responses  to  the  Symbol  palm  computer  in  the 
chemical  gear  condition  are  presented  in  Figure  35.  The  means  and  standard  deviations  are  all  m 
the  acceptable  range  for  the  Symbol  Palm  in  the  chemical  gear  condition. 
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Palm  Computers  -  Symbol  with  Chemical  Gear 


Completely  Reasonably  Borderline  Moderately  Extremely 

Acceptable  Acceptable  Unacceptable  Unacceptable 


Figure  35.  Means  and  Deviations  -  Symbol  Palm  Computer  with  Chemical  Gear 

No  problems  or  potential  problems  for  the  Symbol  Palm  device  were  revealed  through 
user  rankings  in  the  chemical  gear  condition.  User  responses  were  quite  favorable,  and  included 
such  subjective  comments  as:  “AWESOME!”  “EXCELLENT!!”  and  “Just  as  I  expected, 
perfect.”  More  specific  comments  indicated  that  participants  appreciated  the  ease  with  which 
they  could  select  a  field,  that  the  device  appeared  to  be  rugged,  and  that  the  screen  was 
completely  readable  in  direct  sunlight. 

4.3.3  Rankings  for  Palm  Devices 

Statistical  analysis  of  the  rankings  may  show  if  one  palm  device  is  preferred  over  the 
others.  Using  the  x  2  analysis,  the  analysis  is  done  on  the  “Most  Preferred”  rankings  provided  by 
participants.  If  the  distribution  is  equal,  the  expected  value  for  each  configuration  is  6  (n  =  12). 
With  one  (1)  degree  of  freedom  (d f  =  k-1)  and  a  =  .05,  the  critical  value  is  3.8414. 

In  the  non-chemical  gear  condition,  the  calculation  yields  %  2  =  1 .333.  Since  %  2  <  3.8414, 
the  null  hypothesis  cannot  be  rejected  and  therefore  the  data  did  not  provide  significant  evidence 
that  the  subjects  had  a  statistical  preference  for  a  particular  palm  device  in  the  non-chemical  gear 
condition.  In  the  chemical  gear  condition,  the  calculation  yields  %2=  .3333.  Since,  again,  %  < 
3.8414,  the  null  hypothesis  cannot  be  rejected  and  therefore  the  data  did  not  provide  significant 
evidence  that  the  subjects  had  a  statistical  preference  for  a  particular  palm  device  in  the  chemical 
gear  condition.  Even  though  this  factor  is  not  statistically  significant  in  either  condition,  it 
should  be  noted  that  the  distribution  is  not  as  strong  due  to  the  study’s  small  sample  size;  thus, 
other  analysis  methods  were  used  to  evaluate  device  preferences. 

Overall,  visual  inspection  of  ranking  data  for  the  palm  devices  substantiates  findings 
from  the  rating  data;  users  preferred  the  Symbol  device  while  wearing  chemical  gear  and  when 
not  wearing  the  extra  gear  (see  Figure  36). 
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Figure  36.  Overall  Preference  Rankings  for  Palm  Computers 

4.4  Alternative  computers: 

4.4.1  LXE 

The  LXE  alternative  computer  has  a  63 -button  quasi-QWERTY  hard  keypad.  The  input 
device  options  are  the  hard  keyboard,  or  a  touch  screen  that  can  be  accessed  via  the  stylus.  The 
device  weighs  approximately  one  and  one-half  pounds. 

4.4.1. 1  Without  Chemical  Gear 

The  means  and  standard  deviations  of  user  responses  to  the  LXE  in  the  non-chemical 
gear  condition  are  presented  in  Figure  37.  Note  that  the  mean  response  to  the  Reading  Data 
category  on  the  LXE  in  the  non-chemical  gear  condition  is  in  the  unacceptable  range,  indicating 
a  problem  with  this  aspect  of  the  device.  Furthermore,  the  standard  deviations  in  the  Display 
Size,  Keyboard  Layout,  Navigating,  Entering  Data,  and  Drop  Down  Menu  categories  extend  into 
the  unacceptable  range,  suggesting  varying  potential  for  a  problems  with  these  categories  on  the 
LXE. 
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Alternative  Computers  -  LXE 


Completely  Reasonably  Borderline  Moderately  Extremely 
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Figure  37.  Means  and  Deviations  -  LXE  Alternative  Computer 
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Figure  38.  Individual  Ratings  for  LXE  Alternative  Computer 


Participant  responses  during  the  non-chemical  gear  condition  identified  problems  with 
reading  data  on  the  LXE  computer.  Participant  responses  also  identified  potential  for  problems 
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with  the  display  size,  keyboard  layout,  navigating,  entering  data,  and  the  drop  down  menus  with 
the  LXE  computer  (Figure  38). 

Individual  responses  revealed  the  potential  for  problems  with  the  display  size  on  the  LXE 
computer.  While  ten  individuals  rated  the  display  size  on  this  device  in  the  acceptable  range, 
two  participants  rated  the  display  size  on  the  LXE  “Moderately  Unacceptable.”  Only  one 
participant  commented  specifically  about  the  display  size,  simply  remarking  that  his  least 
favorite  feature  of  the  LXE  computer  was  that  the  “screen  was  not  very  big.”  However,  display 
issues  are  revealed  more  clearly  by  the  ratings  for  the  reading  data  category  on  the  LXE. 
Participant  responses  revealed  that  reading  data  on  the  LXE  computer  is  a  problem.  Only  four 
participants  rated  this  aspect  of  the  device  in  the  acceptable  range.  Two  participants  rated 
reading  data  on  the  LXE  “Borderline,”  five  participants  rated  it  “Moderately  Unacceptable,”  and 
one  participant  rated  it  “Extremely  Unacceptable.”  Five  participants  specified  their  inability  to 
read  the  display  in  sunlight  as  a  least  favorite  feature  of  the  device,  and  a  total  of  seven 
participants  remarked  on  their  difficulty  viewing  the  display  in  sunlight;  e.g.,  “display  (was) 
extremely  hard  to  use  in  direct  sunlight”;  “readability  of  the  display,  screen  very  hard  to  see  -  too 
much  glare”;  “the  screen  wasn’t  the  clearest  I’ve  seen”;  “you  can  only  hold  the  unit  a  certain 
way  to  read  the  screen”;  “hard  to  see  top  tool  bar  in  the  sun.” 

Four  areas  related  to  navigating  and  entering  data  on  the  LXE  computer  were  revealed  as 
potential  problems.  Overall  navigation  on  the  LXE  showed  some  potential  for  a  problem;  while 
nine  individuals  rated  navigation  on  the  LXE  computer  in  the  acceptable  range  (with  six  of  those 
providing  a  rating  of  “Completely  Acceptable”),  two  individuals  rated  it  “Borderline,”  and  one 
individual  rated  it  “Extremely  Unacceptable.”  It  is  not  entirely  clear  if  this  navigation  issue  is 
related  to  the  difficulty  viewing  the  display  in  the  sun;  e.g.,  “hard  to  see  top  of  screen.” 

Entering  data  on  the  LXE  computer  was  also  highlighted  as  a  potential  problem  by  user 
ratings;  while  seven  individuals  provided  ratings  of  the  device  in  the  acceptable  range,  three 
participants  rated  entering  data  on  the  LXE  as  “Borderline,”  one  rated  it  “Moderately 
Unacceptable,”  and  one  rated  it  “Extremely  Unacceptable.”  Two  participants  did  indicate  that 
they  appreciated  the  ability  to  enter  data  via  multiple  methods,  specifically,  the  keypad  and  the 
stylus.  All  other  comments  relevant  to  entering  data  on  the  LXE  were  specific  to  the  keyboard 
and  its  layout.  Participant  responses  to  the  keyboard  layout  on  the  LXE  computer  highlight  a 
potential  problem  with  this  area;  although  six  individuals  rated  the  LXE  in  the  acceptable  range 
for  this  category,  four  individuals  rated  the  keyboard  layout  as  “Borderline,”  and  two  participants 
rated  it  “Extremely  Unacceptable.”  Five  participants  specified  some  aspect  of  the 
keyboard/keyboard  layout  as  a  least  favorite  feature  of  the  device;  overall,  seven  individuals 
commented  on  some  difficulty  with  the  keyboard  and  its  layout.  Although  two  individuals 
provided  positive  comments  about  the  keypad  -  “the  keypad  (is)  easy  to  read/use”  -  other 
individuals  clarified  their  difficulties;  a  least  favorite  feature  was  that  “it  was  not  a  traditional 
key  configuration,”  or  the  “setup  of  keys,”  it  had  “no  onscreen  keys,”  it  was  “very  hard  to  use,” 
and  the  “keyboard  layout  takes  time  to  get  used  to.”  One  individual  was  positive  about  the 
ability  to  enter  data  “except  for  the  space  button  -  it  needs  to  be  bigger.  I  keep  wanting  to  hit  the 
enter  button  instead.” 

The  final  area  relevant  to  navigation  wherein  potential  problems  were  implied  is  the 
category  of  Use  of  Drop  Down  Menus  on  the  device.  Eight  participants  provided  ratings  in  the 
acceptable  range.  However,  two  participants  rated  use  of  drop  down  menus  on  the  LXE  as 
“Borderline,”  and  one  participant  rated  this  as  “Extremely  Unacceptable.”  While  no  comments 
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were  provided  that  specifically  corroborates  these  ratings,  screen  visibility  may  have  impacted 
participant  interactions  with  this  feature. 

In  other  general  remarks,  two  participants  specified  that  they  were  pleased  with  the  “good 
size”  of  the  LXE  computer. 

4.4. 1.2  With  Chemical  Gear 

The  means  and  standard  deviations  of  user  responses  to  the  LXE  in  the  chemical  gear 
condition  are  presented  in  Figure  39.  Note  that  the  mean  response  to  the  Reading  Data  category 
on  the  LXE  in  the  chemical  gear  condition  is  in  the  unacceptable  range,  indicating  a  problem 
with  this  aspect  of  the  device.  Furthermore,  the  standard  deviation  in  the  Entering  Data  category 
extends  into  the  unacceptable  range,  suggesting  potential  for  a  problem  with  this  category  on  the 
LXE  in  the  chemical  gear  condition. 
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Figure  39.  Means  and  Deviations  -  LXE  Alternative  Computer  with  Chemical  Gear 
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Figure  40,  Individual  Ratings  for  LXE  Alternative  Computer  with  Chemical  Gear 

Individual  participant  responses  during  the  chemical  gear  condition  further  identified 
problems  with  reading  data,  and  revealed  the  potential  for  problems  with  data  entry,  on  the  LXE 
computer  (Figure  40).  Six  participants  rated  the  device  in  the  acceptable  range  for  the  Entering 
Data  category  in  this  condition.  However,  four  individuals  rated  the  device  as  Borderline,  and 
two  individuals  rated  entering  data  on  the  LXE  in  the  chemical  gear  condition  as  Extremely 
Unacceptable.”  Specifically,  six  participants  indicated  that  they  had  some  difficulty  using  the 
keyboard  due  to  the  small  size  of  the  buttons,  and  three  of  these  listed  the  keys/keypad  as  their 
least  favorite  feature  on  the  device.  “The  keypad  was  hard  to  use  with  chemical  gloves.  If  you 
have  bigger  (thicker)  gloves  on,  you  will  not  be  able  to  hit  the  case  s  hard  keyboard.  It  did 
seem  that  the  keys  were  a  little  close  together.”  “Buttons  too  small  for  gloves. 

Participant  responses  further  indicate  that  there  is  a  problem  reading  data  on  the  LXE  in 
the  chemical  gear  condition.  Only  two  participants  rated  the  device  in  the  acceptable  range  for 
this  category.  Five  participants  rated  the  device  as  “Borderline,’  one  participant  rated  the  device 
as  “Moderately  Unacceptable,”  and  four  participants  rated  the  reading  data  on  the  LXE  as 
“Extremely  Unacceptable.”  Seven  participants  commented  that  it  was  difficult  to  read  the  screen 
in  sunlight  due  to  glare,  and  four  of  these  specified  the  display,  or  the  glare  on  the  display,  as 
their  least  favorite  features  of  the  device.  Some  comments  include:  “the  contrast  between  lights 
and  darks  needs  more  definition”;  “the  shadows  from  my  gloves  made  it  hard  to  see  on  the  small 
screen.  Had  to  move  the  unit  around  a  lot  to  get  a  picture  without  the  shadows  ,  and  extremely 
hard  to  see,  even  without  chem.  gear.” 

Three  participants  remarked  positively  regarding  the  small  size  of  the  LXE  computer. 
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4.4.2  Xolorer 

The  Xplorer  alternative  computer  has  an  onscreen  standard  QWERTY  keyboard.  The 
input  device  option  is  a  touch  screen  that  can  be  accessed  via  the  stylus. 

4.4.2. 1  Without  Chemical  Gear 

The  means  and  standard  deviations  of  user  responses  to  the  Xplorer  alternative  computer 
in  the  non-chemical  gear  condition  are  presented  in  Figure  41.  No  means  are  in  the  unacceptable 
range.  Only  one  category.  Layout,  was  assigned  ratings  wherein  the  standard  deviation  extends 
into  the  unacceptable  range,  indicating  the  potential  for  a  problem  in  this  area. 


Alternative  Computers  -  Xplorer 


Completely  Reasonably  Borderline  Moderately  Extremely 

Acceptable  Acceptable  Unacceptable  Unacceptable 


Figure  41.  Means  and  Deviations  -  Xplorer  Alternative  Computer 
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Figure  42.  Individual  Ratings  for  Xplorer  Alternative  Computer 

Participant  responses  in  the  non-chemical  gear  condition  identified  some  potential  for 
problems  with  the  keyboard  layout  on  the  Xplorer  computer  (Figure  42).  After  using  the  Xplorer 
computer  without  chemical  gear,  eight  participants  rated  the  keyboard  layout  on  the  device  in  the 
acceptable  range.  However,  two  participants  rated  the  Xplorer  keyboard  layout  as  “Borderline,” 
and  two  participants  rated  it  as  “Moderately  Unacceptable.”  Only  one  individual  specified  the 
lack  of  a  hard  keyboard  as  his  least  favorite  feature  on  the  device.  Other  comments  that 
illuminate  any  difficulty  users  may  have  had  with  the  on-screen  keyboard  of  the  Xplorer  include 
the  following:  “When  carrying  into  sun,  bumped  by  knee  and  ended  up  with  extra  characters  on 
screen”;  “keyboard/key  sizes  should  be  bigger. . .  keys  were  small,  I  prefer  using  my  fingers  to 
type  instead  of  stylus”;  “had  to  keep  moving  keyboard  out  of  the  way.”  Four  participants  listed 
the  touch  screen  and/or  its  “full,  standard  keys”  as  their  favorite  feature  on  the  device. 

There  were  very  few  general  comments  regarding  the  Xplorer  computer  during  the  non¬ 
chemical  gear  condition.  Two  individuals  simply  pointed  out  how  much  they  liked  the  device: 
“outstanding  device,  overall  probably  my  favorite”;  “this  device  has  to  be  the  best  one  yet! 

Constructive  comments  regarding  the  Xplorer  computer  indicated  that  two  participants 
had  some  difficulty  with  the  weight  of  the  computer,  noting  that  the  device  was  “definitely  too 
heavy.”  Furthermore,  two  participants  indicated  concern  that  there  was  no  apparent  place  to 
stow  a  pointing  device/stylus  for  the  device. 

4.4.2.2  With  Chemical  Gear 

The  means  and  standard  deviations  of  user  responses  to  the  Xplorer  Slate  in  the  chemical 
gear  condition  are  presented  in  Figure  43.  All  of  the  means  and  standard  deviations  are  in  the 
acceptable  range  for  the  Xplorer  Slate  in  the  chemical  gear  condition. 
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Alternative  Computers  -  Xplorer  with  Chemical  Gear 


Completely  Reasonably  Borderline  Moderately  Extremely 

Acceptable  Acceptable  Unacceptable  Unacceptable 


Figure  43.  Means  and  Deviations  -  Xplorer  Alternative  Computer  with  Chemical  Gear 

There  were  very  few  comments  regarding  the  Xplorer  computer  during  the  chemical  gear 
condition.  Constructive  comments  regarding  the  Xplorer  computer  indicated  that  participants 
had  some  difficulty  with  the  weight  of  the  computer  —  two  participants  noted  that  the  device  was 
too  heavy.  Two  participants  also  commented  that  the  readability  of  the  screen  was  reduced  due 
to  a  glare  that  was  produced  when  using  the  Xplorer  computer  in  direct  sunlight. 
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5  Discussion 


The  purpose  of  this  study  was  to  provide  a  subjective  judgment,  based  on  standard 
usability  testing  methods,  as  to  the  usability  of  various  mobile  devices  on  the  flightline.  Based 
on  participant  comments,  ratings  and  rankings,  the  devices  can  be  categorized  as  follows: 

1.  Usable  as  designed.  Comments,  ratings  and  rankings  revealed  no  problems  or  potential 
problems. 

2.  Usable  as  currently  designed,  but  with  considerations  for  future  designs.  Comments, 
ratings  and  rankings  revealed  potential  problems,  but  no  problems. 

3.  Not  usable  as  currently  designed.  Comments,  ratings  and  rankings  revealed  a  problem  or 
problems  with  the  device  on  the  flightline. 

Using  information  collected  in  the  study,  each  device  has  been  assigned  to  an  appropriate 
category. 

5.1  Laptop  Devices 

5.1.1  Dolch 

The  Dolch  computer  met  usability  requirements  for  performing  data  collection  type  tasks 
on  the  flightline.  The  Dolch  computer  is  usable  as  currently  designed,  but  with  considerations 
for  future  designs.  While,  overall,  users  provided  positive  comments,  ratings  and  rankings  for 
the  device,  it  should  be  noted  that  when  wearing  chemical  gear  there  were  potential  problems 
using  the  mouse  touchpad.  This  was  the  least  favorite  feature  of  the  device.  Fingertip  sensitivity 
when  wearing  chemical  gloves  is  likely  to  be  reduced;  as  such  this  may  impact  the  user’s  ability 
to  adequately  interact  with  the  mouse  touchpad. 

5.1.2  EDNA 

Clearly  the  EDNA  computer  did  not  meet  usability  requirements  for  data  collection  on 
the  flightline.  The  EDNA  Computer  is  not  usable  as  currently  designed.  Even  the  individual 
ranking  data  indicates  that  the  EDNA  was  the  device  least  preferred  by  users,  both  with  and 
without  chemical  gear,  when  compared  with  the  other  laptops.  Real  usability  problems  were 
identified  both  with  and  without  chem.  gear  for  reading  data  and  perceived  portability.  Users 
also  indicated  usability  problems  with  the  size  of  the  device.  Comments  were  explicit  in  terms 
of  the  inability  to  read  the  display  screen,  especially  in  the  sunlight.  In  accordance  with  user 
comments,  this  device  did  not  offer  the  portability  requirements  users  want  to  have  on  the 
flightline.  It  is  important  to  note  that  the  EDNA  was  the  only  device  that  required  a  tethered 
power  cable  in  order  to  operate.  Along  with  the  size  and  weight  of  the  device,  this  umbilical  led 
users  to  rate  the  computer  as  unacceptable  for  portability.  In  addition  to  these  usability 
problems,  many  potential  usability  problems  were  also  indicated  on  the  EDNA.  Six  categories 
were  identified  as  potential  problems  in  the  non-chemical  gear  condition:  Display  Size, 
Keyboard  Layout,  Pointing  Device,  Navigation,  Entering  Data,  and  Dropdown  Menus. 

5.1.3  Itronix  GoBook 

The  Itronix  GoBook  computer  met  usability  requirements  for  performing  data  collection 
type  tasks  on  the  flightline.  The  Itronix  GoBook  is  usable  as  currently  designed,  but  with 
considerations  for  future  designs.  Users  only  indicated  one  potential  problem  when  using  the 
device  without  chemical  gear;  this  was  in  Perceived  Ruggedization.  It  is  important  to  note  that 
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this  is  perceived  ruggedization  not  actual  ruggedization.  Users  believed  that  the  device  was  not 
as  rugged  as  it  should  be.  This  misconception  might  be  overcome  with  training,  familiarization 
or  some  modification  to  the  design.  When  wearing  chemical  gear,  participants  indicated 
potential  problems  Entering  Data.  These  potential  problems  related  to  using  the  pointing  device 
or  mouse  with  chemical  gear.  Fingertip  sensitivity  when  wearing  chemical  gloves  is  likely  to  be 
reduced;  this  may  impact  the  user’s  ability  to  adequately  interact  with  the  mouse. 

5.1.4  Panasonic  Touahbook 

The  Panasonic  Toughbook  computer  met  requirements  for  usability  of  the  device  for 
performing  data  collection  type  tasks  on  the  flightline.  The  Panasonic  Toughbook  is  usable  as 
currently  designed,  but  with  considerations  for  future  designs.  While,  overall,  users  provided 
positive  comments,  ratings  and  rankings  for  the  device,  it  should  be  noted  that  when  wearing 
chemical  gear  there  was  a  potential  usability  problem  when  using  the  Pointing  Device, 
specifically  using  the  mouse.  Fingertip  sensitivity  when  wearing  chemical  gloves  is  likely  to  be 
reduced;  this  may  impact  the  user’s  ability  to  adequately  interact  with  the  mouse. 

5.1.5  Paravant  Scorpion 

The  Paravant  Scorpion  computer  met  requirements  for  usability  of  the  device  for 
performing  data  collection  type  tasks  on  the  flightline.  The  Paravant  Scorpion  is  usable  as 
currently  designed,  but  with  considerations  for  future  designs.  Users  only  indicated  two 
potential  problems  when  using  the  device  without  chemical  gear:  Pointing  Device,  and 
Navigation.  Participants  noted  that  the  mouse  adaptor  was  the  reason  for  the  unacceptable 
ratings  in  the  Pointing  Device  category.  No  problems  were  indicated  when  using  this  device 
with  chemical  gear. 

5.2  Handheld  Devices 

5.2.1  Dolphin  7400 

The  Dolphin  7400  handheld  computer  met  usability  requirements  for  data  collection 
activities  on  the  flightline.  The  Dolphin  7400  is  usable  as  currently  designed,  but  with 
considerations  for  future  designs.  Users  indicated  potential  problems  with  the  Keyboard  Layout 
and  Entering  Data  when  not  wearing  chemical  gear.  This  feedback  is  related  to  the  arrangement 
of  the  keyboard,  an  alphabetical  arrangement,  as  well  as  the  use  of  function  keys  to  input  certain 
characters.  The  on-screen  keyboard,  while  preferred  to  the  hard  keyboard,  was  also  identified  as 
potentially  problematic  by  several  users. 

5.2.2  Intermec  700 

The  Intermec  700  handheld  computer  met  usability  requirements  for  data  collection 
activities  on  the  flightline.  The  Intermec  700  is  usable  as  currently  designed,  but  with 
considerations  for  future  designs.  Users  indicated  potential  problems  with  Entering  Data  and  the 
Keyboard  Layout  when  not  wearing  chemical  gear.  This  feedback  is  related  to  the  arrangement 
of  the  keyboard,  a  telephone  style  arrangement.  This  arrangement  required  users  to  press  keys 
multiple  times  at  a  certain  rate  to  enter  specific  alphabetic  characters.  Users  also  indicated 
potential  problems  Entering  Data  when  wearing  chemical  gear.  Fingertip  sensitivity  while 
wearing  chemical  gear  may  have  contributed  to  this  concern. 


5.3  Palm  Devices 

5.3.1  Palm  515 


The  Palm  515  is  usable  as  currently  designed,  but  with  considerations  for  future  designs. 
The  Palm  515  computer  met  requirements  for  usability  of  the  device  for  performing  data 
collection  type  tasks  on  the  flightline — no  actual  problems  were  identified  when  performing  the 
analysis.  However,  there  were  five  potential  problems  when  not  wearing  the  chemical  gear  and 
one  when  wearing  gear.  This  number  alone  indicates  that  the  device  may  not  be  as  appropriate 
for  flightline  data  collection  as  other  devices  available.  The  five  potential  usability  problems 
were  in  these  areas:  Apparent  Ruggedization,  Pointing  Device,  Navigating,  Entering  Data,  and 
Dropdown  Menus.  In  chemical  gear,  users  indicated  potential  problems  with  the  Pointing 
Device.  Most  of  the  user  comments  related  to  issues  associated  with  the  small  size  of  the  device, 
the  screen,  and  the  seemingly  fragile  nature  of  this  palm  device. 

5.3.2  Symbol 

The  Symbol  computer  met  requirements  for  usability  of  the  device  for  performing  data 
collection  type  tasks  on  the  flightline.  The  Symbol  is  usable  as  designed.  This  is  the  only  device 
included  in  this  evaluation  that  received  ratings  in  the  acceptable  range  for  all  categories,  both 
with  and  without  chemical  gear.  While  participants  used  die  stylus  and  onscreen  keyboard  as  the 
primary  method  for  inputting  data  on  this  device,  they  were  aware  that  handwriting  recognition 
is  available  as  a  feature  on  this  device. 

5.4  Alternative  Devices 

5.4.1  LXE 

The  LXE  computer  did  not  meet  usability  requirements  for  data  collection  on  the 
flightline.  The  LXE  is  not  usable  as  currently  designed.  Only  one  area  was  indicated  as  a  real 
usability  problem  both  with  and  without  chemical  gear;  this  was  in  Reading  Data.  This  problem 
reading  data  was  exacerbated  when  attempting  to  read  the  screen  contents  in  the  sunlight.  In 
addition  to  this  usability  problem,  five  potential  problems  were  also  identified.  These  include 
Display  Size,  Keyboard  Layout,  Navigating,  Entering  Data  and  Dropdown  Menus.  These 
potential  problems,  which  individually  do  not  prohibit  usability  on  the  flightline,  indicate  some 
systemic  problems  with  the  usability  of  the  device. 

5.4.2  Xplorer 

The  Xplorer  computer  met  requirements  for  usability  of  the  device  for  performing  data 
collection  type  tasks  on  the  flightline.  The  Xplorer  is  usable  as  currently  designed,  but  with 
considerations  for  future  designs.  Only  one  potential  usability  problem  was  identified  with  this 
computer:  Keyboard  Layout.  This  device  only  offered  the  on-screen  keyboard.  Users  indicated 
this  as  a  potential  problem  with  the  device. 


57 


6  Recommendations 


Over  the  course  of  the  study,  individuals  articulated  several  general  concerns  relevant  to 
any  consideration  of  e-tools  for  flightline  maintenance  activities.  Behavior  of  the  devices  in  the 
environments  in  which  they  will  be  used  is  important;  devices  should  withstand  heat,  sand  and 
sun  exposure,  and  should  also  retain  their  level  of  accessibility  in  darkness.  Particularly,  screens 
and  keypads  should  have  lighting/backlighting  capabilities  in  order  to  be  easily  navigated  and 
used  in  various  low  lighting  conditions. 

Furthermore,  familiarization  with  any  e-tool  requires  a  change  in  accepted  general  and 
individual  processes.  Users  indicated  that,  while  electronic  access  to  information  could  be 
valuable  to  them,  it  is  equally  important  that  this  access  does  not  simply  add  an  additional  step  to 
an  activity.  They  expressed  concern  that,  after  entering  data  using  an  e-tool,  they  would  still  be 
required  to  interface  with  another  device  to  update  any  information  stores.  Users  expect  that 
access  to  data  using  remote  e-tools  is  immediate,  and  provides  direct  updates.  Unless  the  system 
can  be  updated  (and  provide  updates)  real-time,  the  usability,  and  improvement  to  processes,  is 

low. 

During  the  course  of  the  study,  two  device  characteristics  emerged  as  most  desirable. 
Spontaneous  remarks  by  the  participants  indicated  that  the  touch  screen  was  the  most  popular 
feature  available  across  the  devices.  It  allowed  individuals  to  bypass  difficult,  confusing  or 
simply  undesired  hard  keyboards,  and  directly  interface  with  the  data  being  accessed.  This  was 
particularly  true  when  individuals  were  restricted  by  chemical  gear  gloves,  which  reduced 
accuracy  on  a  keyboard,  and  increased  difficulty  grasping  a  slender  stylus.  While  the  stylus  is 
often  provided  as  one  means  of  accessing  a  touch  screen,  fingertip  access  is  also  a  viable  means 
of  entering  data  via  the  touch  screen.  The  other  feature  commonly  mentioned  as  important  was 
size  of  the  device.  As  maintainers  are  accustomed  to  being  encumbered  by  large  manuals  and 
toolboxes,  a  small  portable  device  for  maintenance  data  collection  is  extremely  desirable. 

Overall,  the  Symbol  device  stands  out  as  the  device  most  usable  for  data  collection  in 
flightline  settings.  Users  indicated  that  this  device  might  also  be  appropriate  for  display  of 
checklist  type  technical  data.  This  is  consistent  with  information  gathered  during  previous 
studies  by  AFRL  (Donahoo,  Gorman,  Kancler,  Quill,  Revels  &  Goddard,  2002).  The  device 
offered  a  small  footprint,  a  stylus  and  touch  screen,  handwriting  recognition,  and  an  on-screen 

keyboard. 

The  only  two  devices  not  recommended  for  use  on  the  flightline  for  data  collection  were 
the  EDNA  and  the  LXE.  Clearly,  the  EDNA  had  many  complex  usability  problems  that  would 
prohibit  use  of  the  device  for  flightline  data  collection  without  major  upgrades  to  all  aspects  of 
the  computer.  The  LXE,  on  the  other  hand,  may  be  suitable  for  flightline  use  if  usability  issues 
associated  with  the  screen  readability  can  be  resolved.  The  recommendation  for  the  LXE  device 
is  to  reevaluate  the  screen  being  used  so  as  to  minimize  problems  reading  the  screen  in  sunlight. 
Consideration  for  other  potential  usability  problems  should  also  be  given  when  making 
modifications  to  future  designs. 

Users  indicated  that  the  mouse  is  not  acceptable  as  the  primary  means  of  entering  data. 
Pointing  devices  on  laptops,  specifically  the  mouse  type  devices,  caused  users  to  note  potential 
usability  problems.  This  was  especially  apparent  when  wearing  chemical  gear.  It  is  important  to 
note  that  the  pointing  device  problem  was  consistently  identified  as  a  potential  problem  with  all 
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laptop  devices  (either  with  chemical  gear  or  without).  The  recommendation  is  that  use  of 
pointing  devices  for  flightline  maintenance  e-tools  be  minimized  as  a  primary  means  of 
providing  pointing  capability  to  users.  Users  indicated  the  preference  for  the  touch  screen  on 
these  computing  devices. 

Given  this  information,  it  should  be  noted  that  certain  input  situations  utilizing  a  touch 
screen  might  require  peripheral  pointing  devices,  such  as  a  stylus.  Some  users  expressed  a 
preference  for  a  stylus  when  using  the  touch  screen  with  chemical  gear,  particularly  for  tasks 
requiring  precision,  such  as  utilizing  handwriting  recognition  or  the  onscreen  keyboard.  Use  of 
the  handwriting  recognition  feature,  or  access  to  the  onscreen  keyboard  when  wearing  chemical 
gear,  may  be  more  difficult  if  only  fingertip  access  to  the  touch  screen  is  available. 

Users  indicated  keyboard  layout  on  the  handheld  devices  as  potentially  problematic.  This 
was  true  for  both  devices.  The  Dolphin  offered  an  alphabetically  arranged  keyboard  and  the 
Intermec  offered  a  telephone  style  keyboard;  both  were  identified  as  having  potential  usability 
problems  without  chemical  gear  on.  This  is  consistent  with  other  findings  on  keyboard 
arrangement;  users  -  even  hunt  and  peck  typists  -  prefer  the  QWERTY  style  keyboard  (arranged 
as  it  is  on  the  standard  keyboard)  arrangement  to  other  arrangements,  such  as  alphabetical 
arrangements.  It  is  important  to  note,  however,  that  users  indicated  potential  problems  with  the 
keyboard  layout  on  the  LXE  device;  this  device  mimicked  a  QWERTY  arrangement,  with  the 
exception  that  the  enter  key,  space  bar,  shift  and  backspace  functions  were  relocated.  User 
concerns  with  the  keyboard  had  to  do  with  the  unusual  placement  of  these  keys.  Finally,  the 
Xplorer  offered  only  the  on-screen  keyboard;  users  did  not  like  this  arrangement  either.  As  a 
keyboard  is  somewhat  essential  to  data  entry  -  especially  when  entering  large  amounts  of  text  - 
the  recommendation  is  that  if  a  hard  keyboard  is  used,  it  should  be  a  standard  QWERTY 
arrangement.  If  an  on-screen  keyboard  can  be  available,  it  should  be  included,  but  it  should  not 

be  the  only  means  of  entering  text. 

As  an  alternative  to  having  a  hard  keyboard  or  on-screen  keyboard,  users  seemed  to 
appreciate  the  handwriting  recognition  feature  available  on  the  Symbol  and  Palm  5 1 5  device. 
This  capability  allowed  users  to  enter  small  amounts  of  text  fairly  quickly  without  the  additional 
real  estate  required  for  a  hard  keyboard  (alphabetical,  QWERTY  or  telephone  style).  The 
recommendation  is  to  include  handwriting  recognition  capabilities  on  small  devices  as  opposed 
to  a  hard  keyboard.  On-screen  keyboards  should  still  be  available  as  an  alternative  to  the 
handwriting  recognition. 

This  usability  test  included  evaluation  of  eleven  mobile  computing  devices  for  collecting 
maintenance  data  in  a  flightline  setting.  Three  usability  categories  for  e-tools  resulted  from  this 
test:  1)  Usable  as  designed,  2)  Usable  as  currently  designed,  but  with  considerations  for  future 
designs,  and  3)  not  usable  as  currently  designed.  With  regard  to  the  first  category,  users 
identified  the  Symbol  palm  device  as  having  no  potential  or  actual  usability  problems  for 
collection  of  data  on  the  flightline.  Clearly  such  a  device  is  acceptable  for  maintenance  data 

collection  on  the  flightline.  Of  the  eleven  devices  tested  in  this  evaluation,  ten  devices  were 

identified  as  having  potential  or  real  usability  problems.  Users  identified  eight  devices  as  having 
potential  usability  problems  either  with  or  without  chemical  gear  (i.e.,  classified  in  the  second 
category).  These  devices  are,  in  their  current  configuration,  acceptable  for  flightline  use; 
however,  the  potential  problems  identified  for  these  devices  should  be  addressed  in  future 
enhancements  to  these  devices.  Finally,  users  identified  two  devices  in  the  third  category,  these 
devices  were  identified  as  having  real  usability  problems.  The  EDNA  computer  had  multip  e 
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actual  problems  and  multiple  potential  problems.  These  problems  and  potential  problems  seem 
to  make  this  device  unacceptable  without  major  redesign  of  the  system.  The  LXE  device  had 
one  real  problem  along  with  several  potential  problems.  Modifications  to  this  device  may  make 
it  suitable  for  flightline  use,  if  care  is  given  to  the  usability  concerns  identified. 
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Appendix  A.  Device  Specifications 

Table  TIT-  Laptop  Computer  Specifications 


Product 


Product  Type 


Platform 


Touch  screen 


Keyboard 


Pointing  Device 


Storage 


DOLCH 


NotePAC 

Plus 


Laptop 

Computer 


MS  Windows 
98  or  2000 


No 


Standard 

87button 

QWERTY 

keyboard 


Touchpad 
with  2  button 
control 


Mobile 
Pentium  III 


128MB- 

512MB 

DRAM 


10GB  Min 
Hard  Drive 


EDNA 


Data  not 
available 


PC  Cards  and 
Expansion  Slots 


2  Type  11/1 
Type  III 
PCMCIA 
slots 


Audio 


Yes 


13.3in.  TFT- 
1024x768 


Itronix 


GoBook 


Laptop 

Computer 


MS  Windows 
98  or  2000 


Yes 


Standard 

QWERTY 

keyboard 


Stylus  to  be 
used  with  the 
touch  screen, 
and  a  touchpad 
with  3  button 
control 


850MHz  Intel 
Celeron 


128  MB-512 
MB  RAM 


20  or  30  GB 
Hard  Drive 


Panasonic 


Touqhbook  28 


MS  Windows 
XP/2000 


Yes 


Standard  87 
button 
QWERTY 
keyboard 


Stylus  to  be 
used  with  touch 
screen,  and  a 
touchpad  with  2 
button  control 


800MHz  Intel 
Pentium  III 


256MB-512MB 

SDRAM 


30  GB  Hard 
Drive 


Paravant 


Laptop  Computer 


MS  Windows 
98/2000/NT/XP 


No 


Standard  88 
button  QWERTY 
keyboard 


Pointing  device 
oystick) 


600MHz  Intel 
Pentium  III 


128MB-512MB 

DRAM 


5GB  or  higher 


12in.  xIOinx 
2.5in. 


1  slot  for  Type  I 
or  Type  II  card, 
External  Type 
III  PC  card 
support  with 
optional 
adapter 


Built  in  RJ-1 1 
and  RJ-45 
jacks  for 
integrated 
fax/modem  and 
Ethernet 


USB 

connector,  9  pin 
serial  port,  15 
pin  external 
video  CRT  port 


Yes 


12.1  in  TFT 
SVGA  Outdoor 
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DOLCH 


EDNA 


Itronix 

Panasonic 

Color  Vue 
display  with 

Anti  Glare  and 
Touch  screen 

transmissive, 
anti-reflective 
TFT  Active 
Matrix  Color 
LCD  with  Touch 
screen 

2XAGP  3D 
Graphics 

Engine  with 
motion 

comDensation 

or12.1in. 
800x600 
(SVGA) 
transflective, 
daylight- 
readable  TFT 
Active  Matrix 
Color  LCD  with 
Touch  screen 

Paravant 


Readable  Display 


Support  for  64 
bit  internal 
graphics 
acceleration 
with  integrated 
8MB  SGRAM 
XGA  through 
VGA  port 


Ruggedization 


Battery  Life 


RF  Comm  and 


Other 

Communications 


Combined 
56  K  Modem 
w/10/00  NIC 


10/100  Ethernet 
Port 


Integrated 
802.11b 
Wireless  LAN 
(Cisco  Aironet, 
Lucent  Orinoco) 


Integrated 

Wireless 

Solutions 


DataTAC, 

Mobitex, 

GSM/GPRS, 

IxRTT/CDMA 


Table  IV.  Handheld  Computer  Specifications 


Product 


Product  Type 


Platform 


Touch  screen 


Keyboard 


Pointing  Device 


CPU 


Memo 


Storage 


Dimension 


Weight 


PC  Cards  and  Expansion  Slots 


Handheld  Products 


Dolphin  7400 


Handheld 


MS  Windows  CE 


Yes  _ 


On  screen  QWERTY  keyboard 
and  (35, 43,  or  56  button) 
alphanumeric  keypad 


Stylus  to  be  used  with  the  touch 
screen  _ 


200MHz.  Intel  StrongArm  RISC 


32MB  RAM 


256MB  with  compact  Flash  card 


Intermec 


700  Series  Mobile  Com 


Handheld 


MS  Pocket  PC  2002 


Yes 


On  screen  QWERTY  keyboard 
and  19  button  alphanumeric 
keypad 


Stylus  to  be  used  with  the  touch 
screen 


206MHz,  Intel  StrongArm  1110 
RISC 


64MB  RAM 


32MB 


3.8in  1/4  VGA  (240  x  320  portrait 
mode)  _ 


EL  backlit,  touch  screen  option 
available  _ 


Ruqgedization 


Battery  Life 


RF  Comm  and 


Other  Communications 


Optional  Integrated  56k  modem 


Scanners 


Monochrome  LCD,  3.8" 


240x320  pixels, 
Electroluminescent  backlight 


touch  screen 


5ft  dro 


8-10  hours 


RS232,  IrDAl.l  (115kbps 


10  Base-T  Ethernet 


LAN:  802.11b  (Wi-Fi  certified 


WAN:  GSM/GPRS,  CDPD,  Data 
TAC,  Mobitex 


Bluetooth  compatible  module 
Integrated  linear  or  PDF417 
laser  scanner 
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Table  V.  Palm  Computer  Specifications 


Product 


Product  Type 


Platform 


Touch  screen 


Keyboard 


Pointing  Device 


CPU 


Memo 


Storage 


Dimension 


Weight 


PC  Cards  and  Expansion  Slots 


Palm  OS  v4.1 


Yes 


On  screen  QWERTY  keyboard 


Stylus  to  be  used  with  the  touch 
screen 


Motorola  Draaonball  VZ  33MHz 


16MB  RAM 


4.5  in  x  3.1  in  x  .5  in 


Symbol 


SPT1800 


Handheld  (Palm  device 


Palm  OS 


Yes  _ 


On  screen  QWERTY  keyboard 


Stylus  to  be  used  with  the  touch 
screen 


Motorola  Draaonball  VZ  33MHz 


8MB  RAM/  4MB  ROM 
16MB  RAM/4MB  ROM  (Batch 
version) 


1  in.  (H)  x  3-5/8  in.  W  x  7  in.  L 


Batch  version:  10.6  oz/300  gm 
(,66lbs);  Wireless  version:  12.2 
oz/346  am  (,76lbs) 


No 

No 

TFT  color  screen  with  touch 
screen,  160x160  pixel  display, 
65,000  colors 

High  contrast,  anti-reflective  160  x 
160  monochrome  LCD  display, 
touch  screen 

Ruggedization 


Battery  Life 


RF  Comm  and 


Other  Communications 


Requires  an  ISP  account  and 
data  enabled  phone  or  modem 


Optional  CDPD,  GSM/GPRS, 
user  accessible  SIM  socket  for 
GSM/GPRS 


Internal  (WLAN),  External 
AN) 


Scanners 


Integrated  miniature  scan 
engine 


Table  VI.  Alternative  Computer  Specifications 


Product 


Product  Type 


Platform 


Touch  screen 


Keyboard 


Pointing  Device 


CPU 


Memo 


Storage 


Dimension 


Weight 


LXE 


MX3  _ _ 


Alternative  device  _ 


ROM-DOS  6.22  _ 


Yes  _ 


Hard  63  button  QWERTY  keypad 


Stylus  to  be  used  with  the  touch 
screen _ _ 


Intel  486  SX  ULP  _ 


4MB  Dynamic  RAM,  8MB 
Compact  Flash 


Xplorer  _ 


GeneSvs  Maximus 


Alternative  device  _ 


MS  Windows  XP  _ 


Yes  _ 


On  screen  QWERTY  keyboard 


Stylus  to  be  used  with  the  touch 
screen  _ 


500MHz  Intel  Pentium  III _ 


8MB  onboard  video  RAM 


PC  Cards  and  Expansion  Slots  1  Type  I,  II  or  I 


8in.  x  6in.  XI  .35in.  _ 


1.9  lbs  (30.8OZ) _ 

2  PCMCIA  v.  2.1  slots,  Left  socket 
-  Type  I  or  II,  Center  socket  - 


Audio 


640  x  240  1/2  VGA  LCD  16 
grayscale,  Transflective 
monochrome  (indoor/outdoor), 
6in.  Viewing  area,  .22  dot  pitch 


12.1  in.  SVGA  (800x600), 
Transflective,  active  matrix  color 
LCD.  touch  screen 


Ruggedization 


Battery  Life 


RF  Comm  and  _ 


Other  Communications 


4ft  drops 


8  hours 


6  hours  _ _ _ 

Factory  integrated  WWAN  CDPD, 
GSM  (GPRS),  GPS, _  _ 


Scanners 


Code  39,  Interleaved  2  Of  5 
Discrete  2  of  5  UPC-A,  UPC- 
E,  code  128,  Plessey, 
Codabar  _ 
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Appendix  B.  Data  Collection  Forms 

IN-BRIEFING 

AFRL  is  researching  technologies  that  will  assist  in  performing  work  at  the  Point  of 
Maintenance  (POMx).  These  technologies  will  include  capabilities  to  display  technical  data, 
open  and  close  work  orders,  and  order  parts.  Several  mobile  computing  devices  will  be  used  to 
provide  the  access  needed  to  complete  your  maintenance  jobs. 

Objective: 

The  objective  of  this  evaluation  focuses  on  rating  the  effectiveness  of  several  hardware 
devices  for  viewing  CAMS  forms  on  the  flight  line. 

Hardware: 

•  Panasonic  Toughbook  Computer 

•  Paravant  Scorpion  Computer 

•  Itronix  GoBook  Computer 

•  Dolch  Computer  System 

•  EDNA  Computer 

•  Intermec  700  Series  Handheld  Computer 

•  Dolphin  7400  Handheld  Computer 

•  LXE  Computer 

•  Xplorer  Slate  Computer 

•  Symbol  Palm 

•  Palm  515 

Scenario: 

As  you  are  using  each  device,  imagine  that  you  are  using  it  on  the  flight  line  to  open  a 
work  order,  order  a  part,  or  close  a  work  order.  Please  respond  to  questions  with  this  idea  in 
mind. 

Task: 

There  are  a  total  of  four  stations  we  would  like  you  to  visit.  Your  clipboard  will  have  the 
order  in  which  you  are  to  visit  the  device  stations  in  the  upper  right  hand  comer.  When  you 
reach  the  device  station,  you  will  be  asked  to  perform  a  common  task. 

After  using  all  of  the  devices  you  can  fill  out  the  questionnaires  attached  to  each 
clipboard.  These  questionnaires  will  help  us  obtain  your  overall  impressions  and  feedback  about 
the  strengths  and  weaknesses  of  the  devices  you  viewed  and  evaluated  today. 
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CONSENT  FORM 

NEW  TECHNOLOGIES  FOR  MAINTENANCE  AND  LOGISTICS  INFORMATION 

SYSTEM  STUDIES 

1.  Nature  and  Purpose:  I  have  been  asked  to  volunteer  to  act  as  a  subject  in  the  research 
project  named  above.  The  purpose  of  this  effort  is  to  examine  screen  readability,  hardware 
packaging,  and  keyboard  use  of  the  following  devices:  the  laptops,  Panasonic  Toughbook,  the 
Paravant  computer,  the  ITRONIX  GoBook,  Dolch  computer,  and  the  EDNA  computer.  The 
handheld  computers  Dolphin  7400,  and  Intermec  700.  Slate  and  LXE  computers  include  the 
LXE  and  Xplorer.  Finally,  handheld  computers  include  the  Symbol,  and  Palm  515.  This 
usability  test  will  consists  of  observations  of  task  performance,  and  a  feedback  questionnaire 
about  the  usability  of  the  device(s).  The  time  requirement  is  for  the  evaluation  of  all  devices  will 
not  exceed  6  hours.  This  research  will  be  conducted  in  the  maintenance  and  logistics  facilities  at 
Nellis  AFB.  Specific  sites  will  be  identified  by  the  supporting  commands  involved.  There  will 
be  approximately  12  subjects  in  this  research  effort. 

2.  Experimental  Procedures:  The  participant  will  interact  with  each  of  the  devices  to 
determine  their  usability.  Participants  will  be  required  to  don  chemical  resistant  gloves  during  a 
portion  of  the  task.  After  each  portion  of  the  task,  the  participant  will  fill  out  a  post-condition 
questionnaire.  When  both  portions  are  completed,  the  participant  will  fill  out  a  posttest 
questionnaire. 

3.  Discomfort  and  Risks:  My  participation  will  not  involve  risks  greater  than  I  encounter 
performing  my  normal  duties. 

4.  Precautions  for  Female  Subjects:  None. 

5.  Benefits:  I  will  not  receive  any  known  medical  benefits  resulting  from  participation  in  this 

experiment.  - 

My  participation  in  this  study  will  help  to  ensure  that  the  application  and  further  development  of 
these  technologies  are  designed  to  meet  my  needs.  The  ultimate  benefit  of  this  project  will  be  to 
make  maintenance  and  logistics  personnel  more  effective  and  make  their  jobs  easier.  The  only 
other  way  to  obtain  the  required  information  would  be  to  conduct  studies  in  a  laboratory  setting 
using  non-maintenance  personnel.  These  people  would  not  be  representative  of  maintenance 
personnel,  and  the  information  gathered  would  not  reflect  the  true  needs  of  maintenance 
personnel.  I  am  encouraged  to  provide  the  experimenter  with  feedback  about  the  experiment  so 
that  my  concerns  can  be  considered  in  future  investigations. 

6.  Alternative:  Choosing  not  to  participate  in  this  study  is  your  alternative  to  volunteering  for  the 
study. 

7.  Entitlements  and  Confidentiality: 


a.  Records  of  my  participation  in  this  study  may  only  be  disclosed  according  to  federal 
law,  including  the  Federal  Privacy  Act,  5  U.S.C.  552a,  and  its  implementing  regulations. 
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b.  I  understand  my  entitlements  to  medical  and  dental  care  and/or  compensation  in  the 
event  of  injury  are  governed  by  federal  laws  and  regulations,  and  that  if  I  desire  further 
information  I  may  contact  the  administrator  of  the  medical  treatment  facility  at  99th  MSS/SGST, 
(702)  653-2778. 

c.  If  an  unanticipated  event  (medical  misadventure)  occurs  during  my  participation  in 
this  study,  I  will  be  informed.  If  I  am  not  competent  at  the  time  to  understand  the  nature  of  the 
event,  such  information  will  be  brought  to  the  attention  of  my  next  of  kin. 

d.  The  decision  to  participate  in  this  research  is  completely  voluntary  on  my  part.  No  one 
has  coerced  or  intimidated  me  into  participating  in  this  program.  I  am  participating  because  I 
want  to.  Captain  Matthew  Goddard,  AFRL/HESR,  DSN  986-4401  or  his  representative  has 
adequately  answered  any  and  all  questions  I  have  about  this  study,  my  participation,  and  the 
procedures  involved.  I  understand  that  Captain  Goddard  or  his  representative  will  be  available  to 
answer  any  questions  concerning  procedure  throughout  this  study.  I  understand  that  if 
significant  new  findings  develop  during  the  course  of  this  research,  which  may  relate  to  my 
decision  to  continue  participation,  I  will  be  informed.  I  further  understand  that  I  may  withdraw 
this  consent  at  any  time  and  discontinue  further  participation  in  this  study  without  prejudice  to 
my  entitlements.  I  also  understand  that  the  medical  monitor  of  this  study  may  terminate  my 
participation  in  this  study  if  she  or  he  feels  this  to  be  in  my  best  interest. 


VOLUNTEER  SIGNATURE 


DATE 


INVESTIGATOR  SIGNATURE 


DATE 


WITNESS  SIGNATURE  da  it 

PRIVACY  ACT  STATEMENT 

Attthoritv:  We  are  requesting  disclosure  of  personal  information,  to  include  your  Social 
Security  Number.  Researchers  are  authorized  to  collect  personal  information  (including  social 
security  numbers)  on  research  subjects  under  The  Privacy  Act-5  USC  552a,  10  USC  55,  10 
USC  8013, 32  CFR  219, 45  CFR  Part  46,  and  EO  9397,  November  1943  (SSN). 

Purpose:  It  is  possible  that  latent  risks  or  injuries  inherent  in  this  experiment  will  not  be 
discovered  until  some  time  in  the  future.  The  purpose  of  collecting  this  information  is  to  aid 
researchers  in  locating  you  at  a  future  date  if  further  disclosures  are  appropriate. 

Routine  Uses:  Information  (including  name  and  SSN)  may  be  furnished  to  Federal,  State  and 
local  agencies  for  any  uses  published  by  the  Air  Force  in  the  Federal  Register,  52  FR  16431,  to 
include,  furtherance  of  the  research  involved  with  this  study  and  to  provide  medical  care. 
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Disclosure:  Disclosure  of  the  requested  information  is  voluntary.  No  adverse  action 
whatsoever  will  be  taken  against  you,  and  no  privilege  will  be  denied  you  based  on  the  fact  you 
do  not  disclose  this  information.  However,  your  participation  in  this  study  may  be  impacted  by 
a  refusal  to  provide  this  information. 


ICD  Distribution:  Original  filed  with  protocol  records  by  PI;  copy  1,  subject;  copy  2,  subject’s 
medical  record  (if  more  than  minimal  risk  protocol.) 
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Pre-Test  Questionnaire 


Subject  Number:  _ 
Date:  / 

Experimenter: 


All  information  will  remain  confidential 


Last  Name:  _ 

First  Name:  _ _  Middle  Initial: _  Rank:  _ 

Squadron: _  AFSC: - 

Time  in  AF  (in  years) _  Time  in  ACC  (in  years) - - 

Current  Aircraft _  Time  on  current  aircraft  (in  years) _ _ 

Other  weapon  systems  worked:  _ _ 

Current  job  title: _ .  Years  experience  at  current  job: - 

Previous  jobs  held: _ _  Years  experience  at  previous  jobs: _ _ 

How  often  do  you  use  CAMS? 

What  actions  do  you  take  in  a  typical  CAMS  session?  e.g.,  (open  wo,  close  wo,  order  parts) 
How  long  does  a  typical  CAMS  session  last? 

When  do  you  have  to  immediately  access  CAMS? 

What  types  of  activities  can  wait  until  the  end  of  your  shift  until  they  are  entered  in  CAMS? 
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Itronix  GoBook  Laptop  Computer 

POST-CONDITION  QUESTIONNAIRE 


7Z 


Feature  most  liked: 
Feature  least  liked: 


Panasonic  ToughBook  Laptop  Computer 

POST-CONDITION  QUESTIONNAIRE 


Feature  most  liked: 
Feature  least  liked: 


Dolch  Laptop  Computer 

POST-CONDITION  QUESTIONNAIRE 
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The  perceived  portability  of  the  device 


Paravant  Laptop  Computer 

POST-CONDITION  QUESTIONNAIRE 


Feature  most  liked: 
Feature  least  liked: 


EDNA  Computer 
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Feature  most  liked: 
Feature  least  liked: 


LXE  Computer 

POST-CONDITION  QUESTIONNAIRE 


Feature  most  liked: 
Feature  least  liked: 


Xplorer  Slate  Computer 

POST-CONDITION  QUESTIONNAIRE 
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Feature  most  liked: 
Feature  least  liked: 


Intermec  710  Handheld  Computer 

POST-CONDITION  QUESTIONNAIRE 


Feature  most  liked: 
Feature  least  liked: 


Dolphin  7400  Handheld  Computer 

POST-CONDITION  QUESTIONNAIRE 
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Feature  most  liked: 
Feature  least  liked: 


Symbol  Palm  Computer 

POST-CONDITION  QUESTIONNAIRE 


The  perceived  portability  of  the  device 


Palm  515  Palm  Computer 

POST-CONDITION  QUESTIONNAIRE 
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Feature  most  liked: 
Feature  least  liked: 


Panasonic  ToughBook  Laptop  Computer 

POST-CHEMGEAR  CONDITION  QUESTIONNAIRE 


Itronic  GoBook  Laptop  Computer 

POST-CHEMGEAR  CONDITION  QUESTIONNAIRE 


Dolch  Laptop  Computer 

POST-CHEMGEAR  CONDITION  QUESTIONNAIRE 
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Intermec  710  Handheld  Computer 

POST-CHEMGEAR  CONDITION  QUESTIONNAIRE 


Xplorer  Slate  Computer 

POST-CHEMGEAR  CONDITION  QUESTIONNAIRE 
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Please  include  any  comments  or  recommendations: 


HANDHELD  POST-TEST  QUESTIONNAIRE 
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Please  include  any  comments  or  recommendations: 
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Please  include  any  comments  or  recommendations: 


PALM  POST-TEST  QUESTIONNAIRE 


LAPTOP  CHEM  GEAR  POST-TEST  QUESTIONNAIRE 
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Please  include  any  comments  or  recommendations: 
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Please  include  any  comments  or  recommendations:  _ 


SLATE  AND  LXE  CHEM  GEAR  POST-TEST  QUESTIONNAIRE 
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Please  include  any  comments  or  recommendations: 


Appendix  C:  Nellis  Informal  Heat  and  Sunlight  Evaluation 

August  2002 
1  Introduction 

The  Air  Force  Directorate  of  Maintenance  (USAF/ILM)  and  the  Standard 
Systems  Group,  Maintenance  Systems  Division  (HQ  SSG/ILM)  sponsored  an  evaluation 
of  eleven  potential  electronic  devices  or  “E-Tools”  for  flightline  use.  The  primary 
purpose  of  this  E-Tool  Ruggedized  Operational  Device  Evaluation  and  Observation 
(RODEO)  was  to  examine  hardware  packaging,  software  user  interface,  and 
environmental  factors  associated  with  the  usability  of  several  potential  Point  of 
Maintenance  (POMX)  E-Tools  for  maintenance  data  collection  on  the  flightline. 
AFRL/HESR  and  the  University  of  Dayton  Research  Institute  (UDRI)  were 
commissioned  by  SSG/ILM  to  independently  and  objectively  perform  the  evaluation. 

The  evaluation  was  conducted  at  the  57th  AGS,  Nellis  AFB,  Nevada,  20-22  August  2002. 

While  the  primary  purpose  of  this  effort  was  to  evaluate  maintainer  feedback  on 
usability  of  the  equipment,  many  vendors  and  other  government  representatives  were 
interested  in  determining  the  durability  of  the  hardware  in  extreme  heat  and  direct 
sunlight.  Hardware  is  normally  tested  using  rigorous  procedures  whereby  numerous  units 
of  the  same  type  are  tested  under  very  controlled  settings.  In  this  way,  problems  unique 
to  one  unit  do  not  adversely  affect  the  results  associated  with  that  hardware  type  (the 
results  are  averaged  over  all  the  units  of  the  same  type).  Participants  at  Nellis,  however, 
were  interested  in  an  informal  evaluation  of  the  devices  and  were  willing  to  compromise 
testing  rigor  for  the  opportunities  presented  by  the  circumstances  and  conditions  at  Nellis 
in  August.  That  is,  many  device  types  were  available  to  test  at  once,  heat  and  sunlight 
were  conducive  to  testing,  and  the  facilities  were  already  available  (due  to  the  usability 
test  underway).  Given  their  knowledge  of  testing  procedures,  AFRL/HESR  and  UDRI 
were  asked  to  conduct  the  informal  heat  and  sunlight  evaluation.  The  results  of  this 
evaluation  are  provided  in  this  document. 

2  Method 

All  vendors  who  provided  hardware  for  the  usability  test  were  approached  to 
determine  whether  they  were  interested  in  participating  in  die  heat  and  sunlight  test.  The 
limitations  of  the  testing  procedure  were  explained  to  all  potential  participants  (e.g.,  the 
fact  that  only  one  unit  would  be  used  instead  of  multiple  units,  therefore,  potentially 
rendering  results  unique  to  the  one  unit  that  are  not  representative  of  the  type  of  hardware 
as  a  whole).  Participation  was  completely  voluntary.  If  vendors  for  the  hardware  were 
unavailable  to  give  permission  for  that  device  to  be  included  in  the  test,  the  hardware  was 
excluded  from  the  test. 
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Hardware  assessed  in  the  informal  heat  and  sunlight  testing  included  the 
following  items: 

•  Dolch,  NotePAC  Plus 

•  EDNA 

•  Itronix,  GoBook 

•  LXE,  MX3 

•  Panasonic,  Toughbook  28 

Prior  to  initiating  the  informal  heat  and  sunlight  evaluation,  all  of  the  devices 
were  used  in  the  usability  test.  At  the  beginning  of  the  day  all  batteries  were  all  folly 
charged;  however,  it  should  be  noted  that  the  heat  endured  and  battery  drain  encountered 
during  the  usability  test  might  have  influenced  the  device  performance  results  of  the  heat 
and  sunlight  evaluation.  During  the  usability  test,  units  were  primarily  kept  in  the  shade 
where  temperatures  were  approximately  100°F.  However,  all  units  were  also  periodically 
taken  into  the  sun,  whereby  temperatures  increased  to  approximately  1 1 5  F.  Alter 
usability  testing  was  complete,  unit  batteries  were  charged  to  attempt  to  rejuvenate  the 
battery  life  prior  to  the  heat  and  sunlight  evaluation. 

A  table  was  set  up  in  the  sunlight  and  was  repositioned  throughout  the  testing 
period  so  that  the  hardware  devices  were  always  in  direct  sunlight.  Devices  were 
arranged  side-by-side  on  the  table  and  screens  were  tilted  up  toward  the  sun.  A  piece  ol 
paper  was  taped  over  the  right  half  of  the  screen  on  each  device  so  as  to  subjectively 
evaluate  the  degree  of  visual  display  degradation  throughout  the  test. 

The  test  lasted  for  a  two-hour  period.  Observations  of  ambient  temperature  and 
ambient  humidity  were  taken  approximately  every  15  minutes  for  the  entire  two-hour 
period.  Measurement  of  screen  temperature  and  display  degradation  began  after  one  hour 
of  testing.  These  measurements  were  also  noted  every  15  minutes  for  the  final  hour. 
Display  degradation  was  determined  by  lifting  the  paper  (without  exposing  the  covered 
side  to  the  sun)  and  subjectively  comparing  screen  resolution  and  readability  on  either 
side  of  the  paper.  Observations  were  recorded  for  each  device. 

3  Results 

Ambient  temperatures  averaged  1 13°F  for  the  two-hour  period,  and  ambient 
humidity  averaged  12%  for  the  same  period.  After  the  first  hour  of  testing,  observations 
of  screen  temperatures  and  display  degradation  were  recorded  every  15  minutes.  Screen 
temperatures  are  depicted  in  Figure  1. 
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Screen  Temperatures 


Figure  1.  Screen  Temperatures 


Observations  of  display  degradation  were  also  recorded  during  the  second  hour  of 
the  testing  period.  Results  of  these  observations  are  shown  in  Table  I. 

Tahie  T.  Screen  Degradation  Results 


Time 

Dolch 

L  auiv  *.• 

EDNA 

Itronix 

LXE 

Panasonic 

Paravant 

2*40 

Some 
flowering 
on  the  half 
of  the 
display 
exposed  to 
sunlight 

None 

None 

1 

1 

None 

1 

1 

2'55 

Entire 
display 
blacked  out 

None 

None 

E 

None 

None 

3:10 

Entire 
display 
blacked  out 

None 

None 

Some 

minor 

darkening 

None 

None 

3:25 

Entire 
display 
blacked  out 

None 

None 

Some 

minor 

darkening 

None 

None 

3:40 

Entire 
display 
blacked  out 

None 

None 

Some 

minor 

darkening 

None 

None 
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4  Discussion  and  Conclusions 

Results  from  this  informal  evaluation  of  the  effect  of  heat  and  sunlight  on  the  six 
devices  indicate  that  four  of  the  six  devices  withstood  the  evaluation  conditions  without 
any  noticeable  change.  Of  these  four,  the  Panasonic,  Paravant,  and  Itronix  devices 
withstood  the  evaluation  without  any  difficulties. 

While  there  was  no  observed  degradation  of  the  EDNA  display  during  the  heat 
and  sunlight  evaluation,  results  from  the  usability  test  had  already  indicated  difficulties 
viewing  the  display.  This  test  indicated  users  had  problems  discerning  the  contents  of 
items  on  the  screen  not  only  while  wearing  chemical  gear,  but  also  when  not  wearing 
chemical  gear  (i.e.,  masks).  Therefore,  while  there  was  no  observed  degradation  m  the 
heat  and  sunlight  evaluation,  users  had  previously  determined  that  the  screen  was  not 
readable  in  sunlight— regardless  of  extensive  exposure  to  temperature  and  sunlight. 

The  LXE,  MX3  device  showed  some  minor  darkening;  however  screen  contents 
were  still  somewhat  identifiable.  This  darkening  effect  may  have  been  due  to  the  nature 
of  the  screen  type:  it  was  the  only  passive  matrix  LCD  screen  included  in  the  test. 

Again,  as  suggested  by  the  results  of  the  usability  test,  users  also  had  problems  reading 
the  contents  of  the  LXE  screen  while  wearing  chemical  gear  and  while  not  wearing 
chemical  gear  (i.e.,  masks).  Therefore,  while  the  screen  only  darkened  slightly  in  the 
heat  and  sunlight  evaluation,  users  had  previously  determined  that  the  screen  was  not 
readable  in  sunlight— regardless  of  extensive  exposure  to  temperature  and  sunlight. 

The  Dolch,  NotePAC  Plus  was  not  able  to  withstand  the  heat  and  sunlight 
evaluation  conditions.  After  one  hour  of  testing  the  screen  began  to  flower  on  die  half  of 
the  screen  not  covered  by  paper.  Within  the  next  15  minutes,  die  entire  screen  blacked 
out  and  subsequently  the  screen  temperature  dropped  substantially  (refer  to  Figure  1). 

The  computer  remained  in  this  state  for  the  remainder  of  the  evaluation  It  is  important  to 
note  that  while  the  Dolch  computer  used  in  this  evaluation  indicated  problems  with  heat 
and  sunlight  exposure,  these  problems  could  have  occurred  due  to  several  factors.  These 
factors  include,  but  are  not  limited  to,  the  following  possibilities:  first,  the  device  may 
not  have  been  representative  of  the  device  population-  that  is,  it  may  have  had  some 
unique  problems  that  other  devices  of  this  type  would  not  have  exhibited  in  a  similar  test, 
second,  the  device  battery  may  have  been  low  due  to  usage  during  the  usability  test  m  the 

morning. 

5  Recommendations 

Given  the  results  of  this  informal  heat  and  sunlight  evaluation  and  the  results  of 
the  usability  test  conducted  during  this  same  time  period,  the  following  recommendations 
are  provided: 

•  It  is  recommended  that  rigorous  testing  of  the  Dolch  NotePAC  Plus  be 

conducted  to  assure  that  this  type  of  computer  will  endure  the  temperature  and 
sunlight  conditions  imposed  by  extensive  use  in  Air  Force  flightline 
environments. 
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•  In  accordance  with  recommendations  from  the  usability  test,  the  EDNA 
computer  had  multiple  actual  problems  and  multiple  potential  problems. 
These  problems  and  potential  problems  seem  to  make  this  device 
unacceptable  without  major  redesign  of  the  system. 

•  In  accordance  with  recommendations  from  the  usability  test,  the  LXE  device 
had  one  real  problem  along  with  several  potential  problems.  Modifications  to 
this  device  may  make  it  suitable  for  flightline  use,  if  care  is  given  to  usability 
concerns,  such  as  screen  readability. 
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